Finn arrived with a Windows mixed reality headset and I was interested in Azure and all we knew was that we wanted to put the two together in some way. The mentors we found at the Microsoft Lounge were the catalyst to this reaction. They were always eager to help as well as chat and hangout when we needed a break from coding. In addition, there were a wide variety of mentors with specific kinds of knowledge, such as David, who had in-depth experience with Azure storage, and Ben, who helped us with in-memory stream processing.

What it does

Heatcloud is a tool that allows traditional business know-how, customer engagement testing, and advertisement prowess to be translated into 3D environments. In today’s industry, developers are innovating with VR, MR, and AR, bringing software into immersive 3D environments. However, as the cutting edge of immersive software continues to progress forward, traditional customer engagement tools have been made obsolete. The usual battery of engagement analysis has little relevance to modern, immersive XR applications. Thus, in order for developers, managers, and advertisers to properly take advantage of this exciting new development, we must improve the way we look at customer engagement. What do people look at in an immersive environment? Where do they like to explore? Where don’t they like to explore? What perspectives are the most rewarding and engaging? Imagine being able to A/B test a house. Heatcloud allows developers and enterprise customers to gauge how users interact with the world around them, from examining small household objects to exploring large buildings. Heatcloud’s unique “vantage point” data allows for marketers to discover the most efficient ways to leverage advertising in virtual experiences. And, in the future, Heatcloud’s standardized voxel data can be used in machine learning datasets to classify user behavior with the advanced pattern recognition of today’s convolutional neural networks.

How we built it

The Windows Mixed Reality platform is a relatively new way for developers to make apps for augmented and virtual reality devices, including the Hololens Developer Kit and the various Mixed Reality head-mounted displays available on the market. Some of the main benefits of the MR ecosystem include the compatibility with lightweight laptops with mobile GPUs, the ability to run existing “flat” Windows 10 UWP software, and the relatively low cost of the hardware. The low entry cost of MR hardware permits a significantly higher rate of adoption, and potentially a greater market share in the VR industry compared to the more expensive competitors.

Unity3D allows for rapid development and prototyping of immersive experiences for the Mixed Reality platform. Microsoft’s own open source toolkit library allows for common functions, such as teleportation and basic interaction, to be abstracted into a generalized XR interface. Thus, we could focus our development time on backend data structures and algorithms.

We save user’s engagement data as a three-dimensional volume of voxels, with an arbitrary resolution and size. Each voxel holds a floating point value, representing the amount of “attention” that point in space has received by users. As users explore the environment, the software raymarches through the volume, incrementing the voxel’s floating point values with the appropriate amount of “attention”. As the user continues to explore the environment, the data is additively accumulated over time, growing the data set with more perspectives and more locations. After enough user data has been collected, a full “heatcloud” is visible and saved. This heatcloud can visually represent hotspots and coldspots in terms of user’s exploration preferences, as well as “hot cones”, which represent favored perspectives and vantage points.

This data is useless if it is thrown away after every session. Thus, Heatcloud allows for rapid backup and syncing with the Azure storage cloud, using Azure’s blob storage services. We chose to store the 3D voxel data as a flattened .CSV, which allows for the client machine to rapidly reconstruct the data into the voxel format. In addition, Azure Cognitive Services prefers .CSV files as training data for the highly extensible CNTK framework, enabling us to extend the functionality of Heatcloud in the future. There was significant technical challenges involved with rapidly and efficiently converting 3D voxel data into a flattened CSV, involving performance considerations and stream processing.

Challenges we ran into

One of the challenges that we had to overcome was selecting which storage system was best suited for holding the voxel data. We originally thought about utilizing SQL, but quickly learned that there was no efficient way to insert the voxel heat map data into a table. We then looked into the other forms of Azure storage including Tables, Blobs, and Cosmos. After extensive research and talks with the Microsoft mentors, it was determined that Azure Blob was the best option for our application.

Efficiently and smoothly converting 3D voxel data into flattened .CSV text data was no small task, and reconstructing stored .CSV files back into usable heat clouds was equally as important. In order to avoid disrupting the test user’s immersive experience, the .CSV conversion process must be asynchronous, allowing for rendering and realtime code to continue processing between “batches” of conversion. In addition, the sheer scale of the conversion requires new ways of processing the information. I initially tried to use simple string concatenation to assemble the .CSV file, but after concatenating 432,000 strings, the performance was untenable. Thus, we went to Ben: for help: using in-memory stream processing allows for much more efficient assembly of the .CSV file. As a result, the program and assemble and transmit 432,000 voxels in just a few seconds, all without halting the immersive experience.

Accomplishments that we are proud of

We think that our biggest accomplishment is how we hit all of our goals without having to make too many compromises. In addition, we stayed ahead of schedule throughout the weekend, keeping track of our objectives and the tasks required to achieve them. Efficient work management allowed us to deliver a finished, polished proof of concept.

What we learned

Kate: I learned just how extensive and easily accessible Microsoft Azure really is. The dashboard is a simple and easy way to access all of the services you wish to use for a project. Between the documentation that Microsoft provides on their services and GitHub samples, integration with Unity was a breeze.

Finn: I learned about unique C# features that allow for more efficient processing of large volumes of data. In addition, I discovered first hand how useful open source projects can be! We had almost lost hope of being able to push our data to Azure from inside Unity, because it required a complex authorization header that we did not know how to create for our REST requests. However, an open source Unity repository came to the rescue, providing an authorization header generation implementation that allowed us to push our data to Azure.

What's next for Heatcloud

The most obvious next step for Heatcloud is to leverage Azure’s powerful Cognitive Services to make actionable decisions with our voxel data. We designed our product to use a volume of float values from the very beginning, because a 3D array of floating point values is the best possible input format for machine learning! Neural networks run on tensors, matrices with a large number of dimensions and values. Our voxel data is an example of a 3D tensor, easily used as training and input data to a convolutional network.

When designing our Azure storage scheme, we chose the data format we did because of the high level of compatibility with Azure’s Custom Neural Toolkit (CNTK), which prefers .CSVs for its training and model data. Thus, we hope to be able to train CNTK to classify regions of our voxel data as hotspots, coldspots, and cones of high attention. In some cases, we predict that our voxel data will be too dense or chaotic for humans to find trends and patterns. Thus, the advanced pattern recognition provided by convolution neural networks may prove to be highly valuable.

Because our product is targeted towards developers and businesses that wish to supply their own 3D environments to analyze, another objective that we would like to accomplish is an environment uploading feature. The product currently is a simple proof of concept that does not include this feature, and thus it is required to bring Heatcloud to market.

Overall, we hope that in the future, other engineers will help contribute to Heatcloud, as it has the potential to improve VR experiences for the entire industry. If analytics tools like Heatcloud are improved and built cooperatively by the developer community as a whole, we will be able to build a better future for VR, MR, and all manners of interactive 3D media.

Built With

Share this project: