Rainbow | Seamless Mixed Reality Begins Here

Rainbow organizes virtual information for real space through shareable mixed reality layers.

Comment

Inspiration

We already live in augmented reality. But the user experience is sorely lacking. In order to access information or connect with others virtually, we rely on smartphones, hiding behind our screens and disconnecting from the real world. We believe that accessing virtual information should be effortless, allowing us to interact naturally with the world around us. As A/R technology improves, this reality is moving closer and closer. But how might we ensure that today's information overload doesn't pollute our physical reality? How do we maintain what is true and what is real in a landscape of ubiquitous digital augmentation? To begin to answer these questions we created Rainbow, a design pattern for shareable mixed reality.

What it does

Rainbow is an A/R solution for organizing virtual information in real spaces. By collecting digital content into layers, Rainbow helps users navigate between several different augmented experiences for one location.

User Scenario If Ben needed to get to the T Station today, he’d have to take out his phone, find the right app, get directions, and keeps his nose pressed to the screen while mentally navigating between the image on his phone and world around him. He’d have to switch apps to check the train schedule or see if there might be a better transit option. But with Rainbow, Ben could simply turn on the City Layer published by the municipal government, containing all the transit and tourist information he would need to navigate the space. This information would sync up with actual landmarks and transit hubs, providing a contextual navigation experience that makes sense. If he needed to interact with other information around him, he could easily switch the layer in his A/R headset.

What is layerable reality? Layers of mixed and augmented reality that can be programmed separately. They act as a virtual layer over a physical space, harnessing web APIs, geo-location, and local IOT devices to enhance one or more objects in that space with virtual elements.

  • One space can have many Layers.
  • Unless programmed specifically, only one pLayer can be viewed by one person at one time.
  • Each Layer has several types of rights: content-production and delivery, interactivity, purchasing, editing, administration etc.
  • Layers can be owned and controlled by private companies/ventures, individuals, governments or NGOs.
  • There can also be common Layers, virtual layers where any user can freely edit, add, subtract, or content or interactivity. Typically these sandbox type layers are controlled through a distributed form of moderation and governance, although anarcho-varieties can exist as well.

Examples of Layer Types

  • Public Layer: Controlled by a municipal government. Provides basic directions to subways, common attractions, public bathrooms, and interactivity with municipal data streams (311, 911 etc.).
  • Public/Private Layer owned: Controlled by local Kendall Square district. Provides information about local businesses (promotions, coupons, hours of operation) and events happening in the area
  • Private Layer owned by MIT: Controlled privately by a local university. This layer can include tips/tricks from MIT students about places, restaurants etc. This pLayer can have separate sub-layers that allow for ad-hoc group communication, private messaging, urban gaming etc.
  • Pop-up Layer: Controlled privately and temporarily by Blizzard Entertainment. A temporary (‘pop-up’) pLayer for an urban gaming experience for users subscribed to Blizzard’s World of Warcraft

How we built it

Rainbow was built using mixed reality technology, Mapbox, and Unity, for Meta’s mixed reality technology. For our prototype, we built an experience that allows users to see different views of an IOT devices recording environmental information and providing an NFC login panel for Kendall Square in Cambridge, MA. The user is able to access two layers of reality: the first one being ‘municipal’, with weather information streaming from the sensors and the second one being ‘game’, with a specific event that only select users can access through the NFC.

First, Rainbow required an understanding of our physical space. We then combined this with geolocation and mapbox to map out space in Unity, including specific markers for smart objects Second, we create a smart object, an internet-of-things box, that shows multiple type of information. What information the user views is dependent on the layer the user is in. Third, we built two layers: a Municipal Layer and a Game Layer For the Municipal Layer, we used real-time weather information to share visually with the user. We created a smart box that works with Arduino to get real-time temperature and humidity information as well as the log-in state of unique NFC chips. This information is then threaded into Unity and can be interacted with in A/R.

For the Game Layer, we created an experience, unlocked only once a specific number of users sign in. We used RFID sensors o that can be accessed by select users Finally, we created seamless switching between layers using C# in Unity as well as icons and the user interface using Adobe Illustrator.

Challenges we ran into

Familiarizing ourselves with all the VR/AR hardwares and using them to create great mixed reality experiences was challenging, but in the end everything worked out well. Creating a believable and powerful demo that captures the product features precisely was hard.

Accomplishments that we're proud of

We are most proud that we can envision a seamlessly mixed reality infrastructure using Meta Vision and Mapbox while implementing IoT devices and blockchain concepts in especially such a short time. Our team members have a background in design, engineering, and business, and all firmly believe in a future where mixed reality would generate more personalized daily experiences and genuine social connections. In two days, we have tested various sensors, experimented with Microsoft HoloLens, Meta Vision, and various geospatial mapping tools to bring on the best-mixed reality experience. It took us a while to figure out all the hardware settings, but it worked out well! We are also proud of the support and trust within our team. From the very start, we’ve set our goal to build an awesome product and to challenge ourselves. And we are so proud we’ve done it!

What's next for Rainbow's Beginning

With Rainbow, we are the creating infrastructure for ubiquitous augmented reality using the Internet of Things, an infrastructure that enables seamless mixed reality experience.

Our top priorities include: *Pushing feasibility

  • How can users interact with each other in mixed reality?
  • How do permissions (read, write access) works in different Layers?
  • How might we integrate blockchain for user identification and authentication and/or registering digital asset intellectual property?
  • How might integrate with other connected devices and enable devices to get smart in the city? ** Further testing desirability and viability **
  • How do different users interact with spaces in mixed reality?
  • What does a revenue generation model?
  • How can we scale our idea through leveraging the existing ecosystem?
Share this project:

Updates