SketchSpace AI

• 

  Homepage - Sketch Workspace

• 

  Result Page

• 

  Refinement Dock

Inspiration

Architectural blueprints and hand-drawn room concepts serve two fundamentally distinct spatial purposes, yet standard generative diffusion engines consistently fail to distinguish between them. When designers or homeowners draw an eye-level perspective sketch of a bedroom, text-to-image models frequently introduce severe distortion or hallucinate walls directly in front of the viewer. Conversely, when uploading a flat, top-down 2D floor plan, diffusion models often suffer from perspective collapse—mapping floor boundaries and room divider lines into vertical shelves, cabinetry, or door frames placed directly against the camera lens.

Moreover, iterative design workflows in conventional image generation remain destructive: prompting for a subtle modification, such as adding a potted plant, typically resamples the latent noise space entirely. This rerolls the entire composition, destroying existing furniture arrangements, lighting angles, and room geometry. We created SketchSpace AI to bridge this visualization gap, transforming both structural 2D architectural blueprints and raw freehand perspective sketches into photorealistic spaces while keeping structural boundaries firmly locked.

What it does

SketchSpace AI provides an end-to-end design environment that converts rough line inputs into photorealistic interior visualizations across two distinct processing routes:

• Dual-Path Spatial Synthesis:
  • Path A (3D Isometric Architectural Cutaway): Automatically detects and extrudes 2D blueprint layouts into detailed top-down axonometric cutaways with realistic wood parquet flooring, natural directional sunlight, and clean wall boundaries.
  • Path B (Eye-Level Candid Photography): Identifies perspective room sketches and renders them as photorealistic interior shots framed by a virtual 35mm lens at eye level.
• Intelligent Geometric Classification: Analyzes line density, perimeter wall enclosures, and boundary structures via OpenCV Hough transforms alongside frontend metadata flags to automatically determine whether an input is a 2D floor plan or an eye-level sketch.
• State-Preserving Conversational Refinement: Allows users to request conversational adjustments (e.g., "add indoor plants in the corner") without resetting the design. Edits are processed through a shared-weight latent refinement pipeline that injects new elements while keeping structural walls and existing furniture arrangements intact.

How we built it

We implemented a decoupled three-tier architecture connecting an interactive user interface, an orchestration backend, and a dedicated GPU inference worker:

1. Frontend Client: Built with React 18, Vite, and Tailwind CSS, featuring an interactive digital canvas supporting both freehand drawing tools and architectural preset templates, complete with an interactive before-and-after split comparison slider.
2. Backend Orchestration: Built on FastAPI, handling request routing, image decoding, payload validation, and fallback mechanisms. We integrated Google Gemini 2.5 Flash for multimodal spatial parsing, identifying layout anchors and relative furniture locations.
3. GPU Inference Engine: Deployed within a remote worker environment connected through a high-throughput ngrok bridge:
   • Base Model: SG161222/Realistic_Vision_V5.1_noVAE for photorealistic interior rendering.
   • Precision Autoencoder: stabilityai/sd-vae-ft-mse integrated directly into the pipeline to eliminate gray haze and restore natural dynamic contrast.
   • Spatial Conditioning: lllyasviel/sd-controlnet-scribble applied with adaptive conditioning scales: (\text{scale} = 0.90) for blueprint boundaries and (\text{scale} = 0.72) for relaxed hand-drawn perspective sketches.
   • Refinement Engine: A shared-weight StableDiffusionImg2ImgPipeline configured to share UNet and VAE weights in memory with the base pipeline, running at a calibrated denoising strength: $$0.38 \le \text{strength} \le 0.44$$ This is combined with lexical token weighting (e.g., (lush green potted plant:1.3)) to materialize requested elements without shifting underlying geometry.

Challenges we ran into

• Perspective Collapse on Blueprint Inputs: In initial tests, 2D floor plan presets collapsed into vertical foreground clutter, interpreting bed lines and room boundaries as wall-mounted shelving and cabinet handles. We solved this by implementing targeted negative prompt embeddings against eye-level viewpoints and enforcing higher conditioning scale adherence alongside isometric cutaway framing.
• The Refinement Denoising Sweet Spot: Finding the boundary between image stability and new object generation in img2img was challenging. At low strengths ((\text{strength} < 0.32)), the model prioritized existing pixels too heavily to introduce new objects. At elevated strengths ((\text{strength} > 0.50)), it altered the room layout and shifted walls. We addressed this by setting the default refinement strength to (0.40) and prepending emphasized prompt tokens at the front of the prompt string to guide the CLIP text encoder without altering room structure.
• Zero-Redundancy VRAM Utilization: Loading independent ControlNet and Img2Img pipelines simultaneously risked Out-Of-Memory (OOM) exceptions. We resolved this by instantiating the Img2Img pipeline using references to the already loaded VAE, UNet, and text encoder modules from the primary pipeline, enabling attention slicing to ensure stable inference within consumer-grade GPU constraints.

Accomplishments that we're proud of

• Autonomous Dual-Mode Routing: Successfully architected a system that accurately transitions between top-down 3D isometric cutaways and eye-level room photography based on input geometry.
• Non-Destructive Inpainting Without Masks: Achieved reliable object additions and material tweaks via conversational prompts without forcing users to manually paint inpainting masks over their canvases.
• Clean Architectural Coherence: Extruded 2D line sketches into photorealistic renders that preserve spatial partitions, doors, windows, and core furniture placements.

What we learned

• How structural edge-conditioning models mathematically interact with camera vantage prompts across contrasting coordinate planes.
• The importance of dedicated fine-tuned VAE decoders in resolving latent compression blur and color-space washing in diffusion pipelines.
• Techniques for balancing conditioning weights and denoising parameters in img2img pipelines to achieve targeted semantic edits while preserving scene layout.

What's next for SketchSpace AI

• Exportable 3D Mesh Generation: Expanding from 2D isometric renders into textured 3D .obj and .gltf assets using Gaussian splatting or depth-to-mesh reconstruction for interactive walkthroughs.
• Precise Bounding-Box Manipulation: Introducing interactive bounding-box placement on the canvas to allow users to specify the exact coordinates where new furniture should appear.
• Direct CAD and BIM Ingestion: Supporting vector .dxf and .dwg file uploads directly from architectural drafting software for instant photorealistic client previews.

Built With

• artificial-intelligence
• computer-vision
• controlnet
• deep-learning
• diffusers
• fastapi
• gemini-api
• generative-ai
• google-gemini
• huggingface
• interior-design
• javascript
• ngrok
• opencv
• pydantic
• python
• pytorch
• react
• stable-diffusion
• tailwind-css
• uvicorn
• vite