3D mesh models are the building blocks of AR/VR/MR/3D virtual experiences.

What it does

User can upload images taken of an object at different viewpoints to the web app, which then processes those images to create a 3D mesh.

For each set of user uploaded images, it also creates a .gif animation using ImageMagick which is displayed along side with the 3D mesh model for comparison. The results are stored into a SQLite database and can be viewed again on the Gallery page.

How I built it

There are two components of this project. One is to create a 3D reconstruction pipeline to process images and create 3D textured mesh. The other is to build a web application to allow users to upload images, call the pipeline, store the computed results into a database, and render the 3D mesh results to the user using Three.js. For the web app, I used Flask as the backend, SQLite as the database, Three.js for the 3D model rendering on the frontend.

Challenges I ran into

Creating perfect 3D mesh models is both an art and robust engineering. Structure-from-Motion pipelines often create holes when there are reflective objects, such as bodies of water or shiny metals. It creates holes when I tried to create a 3D model of my MacBook, which reflects light due to its aluminum body. However, under most circumstances, my pipeline is able to create high-resolution textured 3D mesh models comparable to the results from commercial softwares. Additionally, 3D reconstruction process is very compute-resource intensive, and it can take an hour or two to create a 3D mesh model, even though it is currently running on a compute-optimized Google Cloud C2 compute instance.

Accomplishments that I'm proud of

I am proud of building both the pipeline and the web app successfully. It allow average users who do not have 3D graphics experience or access to expensive commercial 3D modeling software to generate 3D mesh models from images taken on their smartphones. The 3D mesh results are stored into a persistent database which allows users to always be able to see their models on the Gallery page.

What I learned

I learned about the 3D reconstruction process and how to use it. I also learned about the different compute instance services offered by Google Cloud, and it can be expensive to run compute-intensive apps like this one.

What's next for Images2Mesh

Keep adding more features to users!

Share this project: