Viz

Change the world: your imagination is the limit! Retexture anything with Viz.

Comment

Inspiration

Have you ever played with a texture pack in your favorite video game (i.e. Minecraft)? It totally changes the way the entire world feels and looks! What if you could do that in real life? This would completely revolutionize countless virtual experiences across entertainment, education, space planning, and more!

What it does

Simply tell it what you want to see via text prompt, and it will reskin the real world!

Effectively, it uses generative AI to create an "IRL texture pack" on demand, then uses cool algorithms we made ourselves to apply the texture to a 3d scan of the environment. See more about implementation in "How we built it".

Users can explore both the retextured and original scans and hover over surfaces to reveal object names and colored masks. They can toggle masks, isolate objects, and inspect individual parts. There are built-in functionalities for saving different texturings, moving around light sources, and even tweaking the existing textures until they are just right!

Viz exports editable USDZ files, diagnostic OBJ/MTL files, JSON metadata, face labels, and a standalone HTML viewer.

How we built it

We built the processing pipeline in Python, using OpenUSD for USDZ files and NumPy and SciPy for geometry processing.

In order to reskin the world, we need to what to reskin. We achieve this by segmenting a 3d mesh into different objects via a hybrid 2d/3d image segmentation pipeline (see below [1]). After that, reskinning everything is a matter of taking the user's prompt, sending it to CREAO, and using its agents to effectively generate different textures (see below [2]). We then take these textures and apply them to the mesh.

3d segmentation pipeline

3D segmentation in its current state is relatively inaccurate and difficult to work with, but can provide a lot of useful insights that may be hard to glean from purely 2d image segmentation. On the other hand, 2d image segmentation is extremely well-developed and highly accurate, with state-of-the-art models like Meta's SAM3 and YOLO.

We use a hybrid 2d/3d image segmentation architecture to segment the 3d mesh. First, we render multiple 2d image (camera) views of the 3d mesh from various locations on the CPU in parallel. Then, we send these images to a cloud GPU worker on Vast.ai running SAM3, Meta's 2d image segmentation model. What is especially unique about SAM3 is that it is promptable: in order to detect objects, you query it with a specific object to find in the image. If it is there, it segments all of the existing objects in the image. So, we have a list of common objects that one might find in a room that SAM should look for, then once it returns the results from the 2d camera views, we map the 2d image segmentations back onto the original mesh. Then, we combine evidence across different 2d views and use the 3d geometry and texture continuity to group specific parts of the surface into objects. Further checks evaluate object shapes and relationships between objects and their parts. For our use case with 3d surface meshes, this proves to be more accurate than both 2d and 3d segmentation are on their own.

Ultimately, this has two advantages: accurate object segmentation (separation into distinct objects), and semantic object segmentation -- the semantics are already hardcoded as queries to SAM3, meaning we can point to a specific object in our mesh and we know the English word for it. This is extremely important for generating textures.

CREAO agents texturizing pipeline

We make use of two agents, an "art director" agent and a "texture artist" agent. We called each via the CREAO API.

The art director takes the prompt given by the user, as well as the segmentation data generated by our segmentation pipeline and figures out what textures to use for each individual object based on the user's prompt and the objects it is segmenting. It creates a more detailed prompt for each texture that we generate . It then returns this data as a formatted JSON to our app.

After receiving the formatted JSON data, the app calls the texture artist agents in parallel to generate all of the textures described by the art director simultaneously. It makes use of CREAO's integration with Nano Banana image generation, and takes advantage of the fact that we can call multiple agents in parallel to get speed gains in generating the textures.

After the textures are generated, they are returned to our app and fitted to the meshes.

Web viewer

We built the interface with HTML, CSS, JavaScript, and a custom WebGL viewer. Local development and processing ran on our personal laptops (M2 Macbook Pro and HP Spectre), while the NVIDIA RTX Pro 5000 GPU worker handled SAM 3 inference.

Challenges we ran into

Segmentation in general was very troublesome and took up most of our time.

One difficult problem was deciding when fragments belonged to the same object. A single wall could appear as several regions, while separate objects could have similar materials and shapes. We added checks for mesh connectivity, surface orientation, texture continuity, and shared evidence across viewpoints.

Assigning parts to objects required a different set of checks. A cushion can belong to a sofa, but a nearby curtain should remain separate. We combined logical relationships with physical containment, contact, and relative scale.

We also fixed the viewer controls so hiding a segmentation mask keeps the original mesh visible. That lets users inspect the labels without losing the room around them.

Accomplishments that we're proud of

We have a functional and shockingly effective workflow from uploading a scan through segmentation, interactive review, texture generation, visualization, and and export. Viz preserves the source geometry, UVs, and textures while adding object labels and relationships between objects and their parts.

This sets us apart from other ai-powered retexturing software -- they are all done with AI image models and have no coherence to the original scales and cannot be manipulated in 3d.

On our room scans, we even verified that fragmented wall and door surfaces could merge while separate doors and chairs stayed distinct. We also added regression coverage for object assembly, part relationships, exports, and viewer behavior.

We are super proud of our segmentation pipeline, as we got this working in under a day and it works really well -- even beyond what we expected (please ask us about this during the demo!)

What we learned

Recognizing an object in an image is only one step toward identifying it in a 3D scan. Combining views requires careful handling of visibility, overlapping masks, and the original mesh. Geometry helps check whether a proposed object makes physical sense.

We also learned that reviewing the output deserves as much attention as producing it. Original textures, hover labels, and separate controls for objects and parts made it easier to spot mistakes and understand the results.

Furthermore, this project was a valuable learning experience in creating end-to-end apps and also working with agents. Both are relatively new territory for us and we are proud to have learned so much from this project.

What's next for Viz

We want to expand to better hardware. Rigs like the Meta Quest 3 come with built in LiDAR scanners, and we believe that being able to actually see the world around you be reskinned would make this project even more immersive than it already is. We envision that someday, you can pick up your headset and imagine a transformed world -- and as you speak your mind, the world reskins itself before your eyes.

We plan on working further with both interior designers and VR game devs to develop our project further and take its usage and implementation to the next level.

Built With

Share this project:

Updates

Submission history