GatherAR

GatherAR , a real-time collaborative app that connects people globally in an AR environment to collaborate, tinker, create and explore new products. Experience new possibilities in AR with GatherAR.

Comment 1

Inspiration

The potential uses for GatherAR are limitless and inspire creativity and innovation. Imagine being able to connect with artists and designers from around the world to brainstorm and work on projects together in real time. Or, envision being able to attend virtual classes and collaborate with classmates on assignments, no matter where you are located. GatherAR also opens up exciting possibilities for shopping, allowing users to explore and discuss new products with others in Augmented Reality rooms. The app's ability to connect people globally and facilitate real-time collaboration opens up a world of possibilities for personal and professional growth and development.

What it does

GatherAR is an app that allows users to connect with one another in an augmented reality environment and collaborate in real-time. To use the app, users first need to sign up and log in. Once logged in, authorized users can create rooms within the app. These rooms serve as a virtual space where users can communicate with one another and work on projects together.

When a room is created, the host is provided with an invite link that can be shared with others to join the room. This invite link can be sent via email or messaging app, or it can be shared directly with users within GatherAR. Members do not need to be signed in to GatherAR in order to join a room; they can simply use the invite link to access the room.

Once inside the room, users will appear as avatar forms and can communicate with one another in real-time using the app's microphone feature. In addition to communicating with one another, users can also add and edit 3D objects within the room. GatherAR provides various controls for picking and placing pickable objects in the AR space, allowing users to easily manipulate and arrange virtual objects as needed. This can be useful for working on projects, sharing ideas, and exploring new products in an immersive environment.

GatherAR also includes built-in components such as editables and widgets (text editor , videoplayer , sketch etc), which allow members to create interactive 3D content within the room. Labels are text boxes that can be used to display information or instructions to users.

When entering a room in GatherAR, users have the option to calibrate the augmented reality scene in order to properly place the virtual space within their real-world environment.

How we built it

GatherAR is developed using a combination of technologies including

  1. ZappAR Universal SDK for React : For Augmented reality
  2. React Three Fiber : For handling 3D contents
  3. Chakra-UI : For Frontend UI
  4. WebSockets : For realtime data transmission
  5. WebRTC : For realtime media transmission
  6. Nodejs Express: As Backend Service
  7. Firebase : For database and auth
  8. Valtio : For react state management

User authentication and profile management in GatherAR is handled by Firebase, while profile creation is done using nodejs server. The app utilizes mesh network architecture for voice communication and a star network architecture for socket data transmission. Media transfers are handled using WebRTC and data connections are established via WebSockets.

Data is transferred between users as commands using WebSockets, which contain an action type, an object ID and object action data. When a command is received by the server, it updates the 3D scene for all participants in the room. This allows the server to save the state of the scene and keep it consistent for all users. User audio is streamed using WebRTC P2P connection.

The GatherAR server is built using Node.js and Express.js and follows a monolithic architecture. However, as the app grows and becomes more complex, it can become more difficult to maintain and scale. To address this, GatherAR has been modularized, which means it has been broken down into smaller, independent components that can be developed and tested separately.

Users can create rooms within GatherAR and invite others to join them using a unique invite link. Once in a room, users can communicate with one another in real-time using their microphones and can also add and edit 3D objects in the scene. The app also includes built-in components such as clickables and labels, which allow users to create interactive 3D content in the room.

When entering a room, GatherAR provides users with the ability to calibrate the augmented reality scene to properly align the virtual space with their real-world environment. The app also includes various controls that allow users to pick up and place objects in the AR space.

In GatherAR, the movement of the user's avatar is captured using the device orientation API. This API allows the app to track the orientation and position of the user's device in real-time, which is then used to update the avatar's movement and position in the augmented reality scene.

Challenges we ran into

At first, there were issues with CORS that needed to be resolved.

One of the main challenges in building GatherAR finding a way to track the coordinates of the world tracker in Zappar. I eventually came up with a solution by creating a tracker that calculates the position of the avatars based on the orientation of the user's phone. The ZappAR Camera is static and i wanted to get the camera position, so I had to find a solution to simulate the camera transforms. To do this, I created a reference mesh at the center of the scene and used its transforms to calculate the camera position based on the orientation of the user's phone. This allows to track and update the movement of avatars in real-time for all users in the room. Another challenge was the latency in communication, and the framerate which was eventually optimized in production builds. Academic schedule also made it difficult to allocate enough time to work on the project. Additionally, there were issues with deployment that had to be addressed. Another one I faced during the development of GatherAR was storing the editing state of the room in the cloud. However, I have temporarily disabled this feature as my servers are currently running on a free tier. Enabling this feature may potentially cause server issues due to the high volume of requests."

Accomplishments that we're proud of

Successfully integrating various technologies, such as React, Zappar Universal, and WebSockets, to build a functional and immersive app.

Implementing user authentication and profile management using Firebase, and designing and implementing a server for profile creation. Using different network architectures, such as mesh and star, to enable voice communication and data transfer, respectively. Implementing the command pattern to improve scalability and maintain consistent state in the app. Modularizing the app to make it easier to maintain and update as it grows. Overcoming technical challenges, such as CORS issues and latency in communication, to successfully build and deploy the app.

What we learned

Learnt how to integrate Zappar Universal to enable augmented reality functionality.

Also gained experience with WebSockets and WebRTC for establishing data connections and media transfers, respectively, software testing , deployment, learned about user authentication and profile management using Firebase, and learnt how to design and implement a server for profile creation. Got some knowledge about different network architectures, such as mesh and star, and how to use them for voice communication and data transfer, respectively. Various design patterns and their benefits for scalability and maintaining a consistent state in the app.

What's next for GatherAR

GatherAR initially created as a web2 application with the goal of serving as an entry point to the metaverse, has the potential to evolve into a highly feasible metaverse software through its utilization of ZapWorks. As we continue to develop and improve GatherAR, my goal is to create a comprehensive platform for personal and professional growth and development within the metaverse.

  1. Optimizing data transmission between peers is an important goal for GatherAR, as it can help improve the overall user experience. By optimizing data transmission, GatherAR can ensure that users are able to seamlessly collaborate and interact with one another in real time without experiencing any delays or issues.

  2. A modular environment with extensions such as a room creator and custom game controls can enhance the versatility and customization options available to GatherAR users. The room creator extension, for example, could allow users to easily create and customize their own augmented rooms for specific purposes, such as project collaboration or socializing. Custom game controls could give users the ability to create and customize their own controls for games or other interactive experiences within GatherAR. Another example is , users could start a augmented store to showcase and buy products by using extensions available in GatherAR marketplace.

  3. Developing design tools can increase productivity for GatherAR users by providing them with a range of tools and features to streamline their workflow. These tools could include features such as templates, shortcuts, and collaboration tools, all of which can help users create and collaborate more efficiently.

  4. Utilizing Zapbox controllers can enhance the immersive augmented reality experience for GatherAR users by allowing them to interact with virtual objects and environments in a more realistic and intuitive way. By using physical controllers to interact with virtual objects, users can feel more connected to the augmented reality environment and have a more engaging and immersive experience.

  5. An asset store for buying and selling digitally created 3D goods can be a valuable resource for GatherAR users, as it allows them to easily access a range of digital assets that they can use in their projects or sell to others.

  6. Connecting to real-world services, such as 3D printing, can be a powerful feature for GatherAR users as it allows them to turn their virtual designs into physical products. By accessing a network of 3D printers located around the world, users can bring their virtual creations to life and have them shipped to them. This can be a valuable resource for artists, designers, and anyone looking to bring their virtual ideas into the physical world.

  7. Custom event management rooms can be a useful feature for GatherAR users who want to invite a limited group of people to specific events or meetings. These rooms can be customized and controlled by the event host, allowing them to easily manage and communicate with attendees.

  8. A realistic avatar creator that generates avatars based on photos of users' faces can add an extra level of personalization and immersion for GatherAR users. By creating avatars that closely resemble their appearance, users can feel more connected to their virtual identity and have a more realistic and immersive experience within GatherAR.

I'm working to implement the Editor in ECS architecture. Although i could'nt finish this within this time frame , i found the potential of scalability

Resources Used

Avatar 3d Model: https://skfb.ly/6WXxV

Built With

Share this project:

Updates