Invision

Inspiration

We were looking for an idea we considered "cool" that was new and had potential to be applied in a variety of applications. A research paper published about a month ago called 3D Gaussian Splatting for Real-Time Radiance Field Rendering piqued our interest. It describes a new way of 3D modeling that is extremely fast at creating high-quality images at the cost of not generating a surface model: perfect for interactive media.

What it does

Our website produces a 3D scene by analyzing the frames of a 2D video capturing an environment. The user can upload their video to our website from their phone and choose the number of iterations to run, which is a tradeoff between training time and quality. They can then view the environment recorded in angles or positions that were not a part of the original data set in real time.

How we built it

Our website has two key components: the backend server and the streaming server.

When a user uploads a video, our server extracts every 10th individual frame and stores that. By doing so, we can collect hundreds of pictures of an object without having the user take that many pictures individually.

Then, another application, COLMAP, trains a one-layer neural network on these images with the number of iterations that can be determined by the user. More iterations take longer but provide an output with sharper quality and less artifacts. Less iterations run quickly but may not produce the desired quality. The final output is a file in stanford triangle format that can be accessed by a server that streams the output directly onto your computer. The server runs an NVIDIA A6000 graphics card and streams the output of the 3D render directly onto the user’s browser through a webRTC connection.

Challenges we ran into

Running the 3DGS rendering code The 3DGS code included in the Github linked to the paper uses Pytorch, which only supports up to CUDA 11.8. It also requires PyNVCodec, which needs a base of CUDA 12.2. We tried using symlinks, nightly releases, Docker, and multiple servers to no avail. In the end, we were able to finally get both running in two different virtual environments. Streaming the 3DGS output The 3DGS software included in the Github generates a unique, multi-file output that cannot be converted to any standard 3D format. Originally, we had set up an Apache Guacamole remote desktop solution for users to stream the output from a remote Windows desktop running compatible streaming software, but after much tinkering, we were able to get a stream of the output running directly with Dylan Ebert’s base code: https://github.com/dylanebert/gaussian-viewer

Accomplishments that we're proud of

We're proud we understand how this brand new technology works despite 3DGS appearing extremely unapproachable when we first started diving into it.

There is no single tutorial utilizing this technology that works out of the box, and our final product synthesizes the lessons learned from two dozen hours of tinkering and experimentation.

What we learned

1. What Gaussians are in the context of rendering.
2. The pros and cons of 3DGS compared to NeRF, the previous iteration of 3D scanning. 3DGS is both much faster and higher quality, but unlike NeRF, it lacks a 3D surface model, so it can't be used in 3D creation suites such as Blender.
3. 3D scanning is being built open-source everyday. It was interesting to see how many of the blog posts or Tweets we used to research and understand 3DGS were posted mere days ago!
4. Software and infrastructure compatibility is a huge pain point: on top of dealing with environment conflicts, we dealt with a dead GPU, which was extremely hard to pinpoint.
5. Learned what hacking really means: researching literature, piecing together pieces of code from various websites and dealing with bugs.

What's next for InVision

Now that we have built a website for scanning with 3DGS as a proof of concept, we can think about other essential features. For example, we could add additional machines so that our website can process multiple videos at a time. Or, we could make an account system and establish a queue. Once we've established the basics, we can InVision an incredible future of applications.

1. Visualizing the real world through aerial drone mapping and 3D satellite data
2. Capturing events and everyday life in 3D rather than 2D photos, breathing life into our archives of the past.
3. Related to the above, walking through massive VR environments as if you were truly there, in a range rather than in a predefined set of points.
4. Planning scenes, camera angles, and movements in media production
5. Quickly capturing real-life environments to use as detailed backgrounds that adjust with player perspective in video game development

We can work on adding features dedicated to these niches as we determine which applications we wish to cater to.

Built With

• fastapi
• python
• pytorch