Mitosis

Inspiration

Online exploration experiences and videogames with user-generated content often suffer from a long initial load time since they have to download all textures in the level to provide a good user experience. If people are going through multiple levels/worlds in a group, those with slower internet often get to spend very little time actually exploring since they spend most of their time in a loading screen.

What it does

We divide textures up into chunks and stream them in realtime across the network, avoiding the initial download process and significantly improving the user experience. However, since the internet is so slow compared to storage devices, we have to work hard to prioritise the textures that make the biggest difference to the scene.

How we built it

We took an existing C# Vulkan rendering engine and added our network texture streaming atop. On the server side, it has a custom ASP.NET Core backend which takes user-uploaded texture files, generates mipmaps, splits them into tiles and then block compresses them to the BC7 format. These chunks are then served to the client over HTTP. On the client, when rendering we use a storage buffer to write feedback information on what textures it wants to use, what areas are available, and the area on the screen they take up. From these we can determine exactly what parts of the texture we want to download, making the most of our limited bandwidth.

Challenges we ran into

oh god so many. We had to update our teams' laptops to have the correct version of .NET Core, which managed to break each of them in a unique way (one ran out of disk space and corrupted the update, one had a broken OpenSSL config which is somehow required for WiFi to work at all, and one lost all its input drivers). Texture uploading in Vulkan is a synchronisation nightmare - we have to download a region of the texture, make the region writeable, copy data onto a CPU-visible buffer on the GPU, copy from the buffer to the texture, and then tell GPU-side code that higher quality texture data is available, while taking into account that all these parts run in parallel and can't write over each other. As a result, we also didn't know if it would work at all until the thing was finished. We ran into a big issue when we realised the server was compressing to KTX 1, and the client was expecting KTX 2!

Accomplishments that we're proud of

The fact that we managed to get so much working in such a little amount of time, while facing challenges from borked computers, misbehaving server configs and at times painfully slow internet :) While some parts of the project we were familiar with from the start, we learned a lot along the way about the specifics of texture loading, compression, and asynchronous I/O.

What we learned

Don't try and update in the middle of a hackathon. Just don't. Save your sanity.

What's next for Mitosis

We hope to continue iterating on it, and have a number of ideas on how to improve performance and make it more useful (e.g. being able to support bigger textures, and implementing extra indirection so we can free up memory for unloaded texture chunks).