mesh by mesh

Inspiration

Learning 3D modeling can be incredibly intimidating. Beginners often stare at a blank viewport, unsure how to translate a reference photo into actual geometry. We wanted to bridge the gap between AI generation and actual human skill acquisition. Instead of just having an AI spit out a finished 3D model that the user learns nothing from, we asked: What if the AI could generate the target, break it down, and then literally hold the user's hand while they build it themselves? We were inspired to build an educational companion that completely respects user privacy and runs with zero infrastructure costs.

What it does

Mesh by Mesh is a Blender 4.2+ addon that serves as a viewport-aware AI tutor. You simply drag and drop a reference photo directly into the Blender viewport. The addon automatically converts that photo into a 3D mesh and intelligently segments it into 5 to 10 color-coded parts.

From there, it acts as an interactive tutor. An embedded LLM generates a structured, step-by-step walkthrough for each colored part, providing viewport annotation overlays and starter directives. The AI actually "sees" your viewport—if you get stuck, it can highlight the specific geometry you need to edit, fix mode-mismatches (like forgetting to enter Edit Mode), or even take over and complete a difficult step for you on demand.

How we built it

We built this to be incredibly lean, relying entirely on a $0-infrastructure, serverless stack with no databases, no accounts, and no telemetry.

Frontend: Blender 4.2+, utilizing Python 3.11, the bpy/bmesh/GPU modules, and Pillow.

3D AI Pipeline: We used Tencent Hunyuan3D-2 for image-to-mesh generation and htm26 Stirnix for segmentation. These run on HuggingFace Spaces and are called asynchronously via gradio_client.

Chat AI: Powered by either Anthropic Claude (Sonnet 4.6 / Opus 4.7 / Haiku 4.5) via their official SDK, or Google Gemini (2.5 Flash / 2.0 Flash) via direct REST calls. Authentication is handled entirely via user-supplied API keys stored safely in Blender Preferences.

Storage: 100% local. State is saved to Blender scene properties, and generated GLBs are cached to ~/Documents/AITutor/.

Challenges we ran into

Integrating cutting-edge AI directly into Blender's strict ecosystem threw a few massive curveballs at us:

Blender 4.2 Quirks: Blender 4.2 has a broken Python FileHandler dispatch for images. We had to engineer a workaround using a depsgraph_update_post hook to auto-capture dropped images into our reference list.

Google SDK Crashes: We initially tried using Google's official Python SDK for Gemini, but discovered its cryptography DLL crashes on many Windows Blender installations. We bypassed this entirely by writing our own direct REST API integration for Gemini.

AI Cold Starts: HuggingFace Spaces can take 60-90 seconds to wake up. We solved this by pre-warming the Space the moment the user enables the addon, cutting generation time down to ~30 seconds.

Messy Topology: Segmenting AI-generated meshes often leaves floating artifacts. We wrote a custom per-material flood-fill algorithm that drops disconnected "particle" islands, ensuring each tutorial part is one clean, continuous piece.

Accomplishments that we're proud of

Zero Server Costs: We successfully engineered a powerful, full-stack AI application that costs us absolutely nothing to host, while keeping user data entirely local.

True Viewport Awareness: Building an AI that doesn't just give text advice, but can actually highlight geometry and correct user states (like mode-mismatches) natively within the 3D view.

Seamless UX: Creating a workflow where a user can just drop a photo into the scene and immediately start an interactive tutorial without leaving Blender.

What we learned

We gained a deep understanding of Blender's internal Python API, specifically mastering the bmesh and GPU modules to render custom interactive overlays. We also learned a lot about prompt engineering for spatial tasks getting an LLM to consistently output structured, chained JSON sub-steps that accurately map to 3D coordinate space and Blender's specific toolset was a massive learning curve.