SketchForge

Autonomous hand-drawn sketch-to-full-stack app generator with OpenCV 5, AWS Graviton & closed-loop visual repair.

Comment

Inspiration

Turning hand-drawn sketches into working applications takes time. Designers spend 4 to 8 hours recreating wireframes in Figma, while developers may need 1 to 2 days to build the frontend, backend, and database. SketchForge automates this process by converting smartphone photos of sketches into accessible, full-stack applications in under 2 seconds.

What it does

SketchForge recognizes UI components, generates React 18 + TypeScript frontend code and a Flask + SQLite backend, then uses browser screenshots and OpenCV 5 to detect visual differences and automatically repair the code.

How we built it

We used OpenCV 5 for perspective correction, shadow removal, grid-line suppression, and component detection. AWS Graviton with COOL accelerates image processing, while Playwright and OpenCV power an autonomous visual verification loop. We also built a browser-based Web Studio and a PySide6 desktop workbench.

Challenges we faced

Uneven lighting and notebook grid lines made sketch recognition difficult. We addressed these issues with morphological filters and FFT-based noise suppression. We also developed a fault-diagnosis engine that translates visual mismatches into targeted CSS and DOM fixes.

Accomplishments

  • Accuracy: F1 score of 0.9727 across 63 test layouts.
  • Speed: Processes a 1240×900 sketch in 156.55 ms.
  • Cost efficiency: AWS Graviton benchmarks showed 40% lower costs and 66.7% more sketches processed per dollar than standard x86 instances.
  • Visual verification: Automated repairs target a visual IoU of at least 0.958.
  • Accessibility: Enforces WCAG AA color contrast of at least 4.5:1.

What we learned

Combining deterministic computer vision with AI-driven code repair makes application generation more predictable. Arm NEON optimization also improved image-processing performance on AWS Graviton.

What's next

We plan to support multi-screen navigation from storyboard sketches, bidirectional Figma and Penpot integration, and live application updates from webcam video.

Built With

Share this project:

Updates

Submission history