We came into this hack eager to build a product in AR, as we believe it’s an emerging technology that will become a key component of future innovation. However, we were unsure about what use case would be suitable for augmented reality. Our idea came when we considered how the modern officespace might be enhanced through AR.
What it does
Our iOS app creates a virtual desktop in augmented reality that can be extended to other users to create a shared workspace. Our desktop includes sticky notes, images, videos, web shortcuts and timers that can be placed and shared in AR.
How we built it
The tech stack revolves around the use of Vuforia, a package for building Augmented Reality applications in Unity.
However, where our project really shines is in the sheer amount of work done from scratch. In terms of the User Interface, our app incorporates complex use of UI elements to bring life to multiple features - such as creating virtual notes, photos, videos, web-links and timers in a workspace. Our app brings to life a collaborative environment for multiple users, who can subscribe to form a peer to peer connection to support real-time collaboration. Similar to the UI elements - the networking protocols, included with sophisticated synchronisation and consensus protocols, were built completely from scratch.
In summary, although the use of augmented reality might seem like the highlight of our application - our main efforts were put into making our app useful, look fantastic and perform seamlessly at the same time!
We wanted to focus specifically on the P2P network stack that we managed to build. The following section outlines how the algorithm works, step by step:
- Each device connects to a network by using the IP address of a peer in the network.
- After 2 peers are paired, a communication channel is built between them.
- Special events in either peer trigger events, which sends updated data about the Game Objects, which update Game Objects in the other peer.
- The Game Object data is first serialized and transmitted using TCP/IP protocol.
- Each peer contains specialised Data Structures for maintaining synchronisation and consensus across each peer.
- However, the consensus protocol is very naive and assumes dirty reads do not occur during the demo.
- All the above maintains real time collaboration between each peer.
Challenges we ran into
- Networking issues
- Synchronisation problems
- ‘Snapping’ features
- Collision difficulties
Accomplishments that we are proud of
- Sharing notes over a peer to peer network. Unity has unfortunately deprecated most of the High Level APIs for Networking since 2018. However, setting up a Peer2Peer network would have been quite difficult, even if the high level APIs were available to us. Therefore, creating the entire networking stack was an enormous success in our part.
- Multiple app features, e.g. virtual notes, photos, videos, alarms, etc. In particular, we were quite satisfied with the overall look and feel of the app. The overall ease of use and applicability of our app was arguably our biggest success.
What we learned
- The power of well-organised teamwork
- Delegation of responsibility
- The advantages of Unity
- Networking Protocols
What's next for collabAR
- Scale our app to support more sophisticated UI and networking features
- Publish our app into the AppStore for revenue
- Vuforia Augmented Reality