Sketch2Model

Home page
Draw the designs
Design is Ready
Estimated cost
Estimation is ready!!

Inspiration

Many people can visualize their dream homes but struggle to translate those ideas into tangible plans or realistic costs. We wanted to build an AI system that bridges creativity and construction, enabling anyone—without design or engineering experience—to sketch or describe their dream house and instantly see a 3D model with a cost estimate.
Our inspiration came from observing how fragmented the design-to-build pipeline is. With the power of multi-agent AI systems, we envisioned an autonomous ecosystem where specialized agents collaborate to make architectural design accessible, visual, and intelligent.

What it does

Sketch2Model turns your hand-drawn sketches or voice descriptions into 3D house designs and provides AI-driven cost estimations.
It uses a multi-agent system where:

🎨 The GeneratorAgent converts sketches or transcribed voice inputs into 3D renders.
💸 The EstimatorAgent predicts material needs and calculates costs using real-time market prices.
🎙️ The VoiceAgent processes spoken descriptions into text.
🧠 The RootAgent orchestrates the entire workflow.

The result is an interactive 3D visualization and a comprehensive cost breakdown covering materials, labor, and structure details — all powered by Claude Sonnet 4 Vision, Fish Audio, and Fetch.ai.

How we built it

Frontend: Built with React 18 + Vite, using the Canvas API for drawing and MediaRecorder API for capturing voice input.
Backend: Developed with FastAPI (Python) and uAgents (Fetch.ai) for multi-agent coordination.
AI & APIs:
- Anthropic Claude Sonnet 4 Vision → for sketch analysis, structure understanding, and cost reasoning.
- Fish Audio API → for speech-to-text transcription.
- Pollinations AI → for generating photorealistic 3D house renders.
Integration: Agents communicate asynchronously through Fetch.ai protocols, with the RootAgent managing workflow and message passing.
Deployment:
- Backend on Render.com
- Frontend on Netlify
- Total operational cost: ~$0–7/month + API usage

Challenges we ran into

Getting accurate 3D reconstruction from incomplete or abstract sketches.
Synchronizing multi-agent communication between Fetch.ai agents using the Chat Protocol.
Handling API rate limits and authentication errors (401/404) from external AI APIs.
Balancing latency—especially when multiple agents were processing vision and pricing tasks simultaneously.
Normalizing real-world market data to maintain accurate cost predictions.

Accomplishments that we're proud of

Built and deployed a fully functional multi-agent architecture that autonomously converts sketches or voice into 3D visualizations.
Achieved real-time cost estimation with breakdowns by category (structural, finishing, roofing, fixtures, labor).
Created an intuitive interactive drawing and voice interface accessible to anyone.
Integrated four independent AI APIs into one cohesive, intelligent workflow.
Deployed a system that’s scalable, cost-efficient, and production-ready.

What we learned

How to design and implement multi-agent AI ecosystems using Fetch.ai.
Deepened our understanding of vision-language models (VLMs) like Claude Sonnet 4 for structure understanding.
Learned to optimize API chaining and parallel execution for faster response times.
Gained experience in merging speech, vision, and cost intelligence into a single unified pipeline.
Understood the importance of user-centric UI/UX when presenting complex AI-driven outputs.

What's next for Sketch2Model

Integrate AR visualization so users can “walk through” their generated houses.
Add sustainability analysis, estimating eco-cost and carbon footprint of materials.
Expand database with global market data for international cost estimation.
Implement contractor matching to connect users with real-world builders.
Introduce federated learning for privacy-preserving improvement of the estimation models.
Optimize multi-agent orchestration with on-chain execution via Fetch.ai for full transparency and automation.