OmniConstruct: Vision-to-Live Architect

OmniConstruct: Gemini Live interprets UI sketches into functional GCP apps via FastAPI & Cloud Run with self-healing AI logic.
Initial hand-drawn UI sketch used as multimodal input for the Gemini Live API to interpret layout and components.
The final live-rendered web app on Google Cloud Run, transformed from a sketch into interactive Tailwind CSS code.

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for OmniConstruct: Vision-to-Live Architect

Inspiration Modern development friction slows down the creative spark. Often, an idea is lost between the "napkin sketch" and the first line of code. I built OmniConstruct AI to act as a Senior Cloud Architect that watches you draw and deploys your vision to Google Cloud in real-time.

What it does OmniConstruct uses the Gemini Live API to observe a physical sketchpad or digital canvas. It identifies UI elements (buttons, inputs, layouts) as you draw them and immediately generates production-ready code, deploying it directly to a live URL.

How we built it Frontend: A React interface streaming camera feed via Gemini Live API.

Backend: A Python (FastAPI) orchestrator on Google Cloud Run.

Intelligence: Gemini analyzes visual intent and maps it to Tailwind CSS templates.

Deployment: Automated pipelines through Google Cloud Build for instant hosting.

Challenges we ran into Handling visual ambiguity was the biggest hurdle. To solve this, I leveraged the Live API's interruption feature. If Gemini is unsure about a component, it verbally asks for clarification, ensuring 100% accuracy before deployment.

Accomplishments that we're proud of We successfully created a "Self-Healing" loop where the agent monitors its own deployment logs on GCP. If a build fails, it auto-corrects the code and re-deploys without human intervention.

What we learned I learned to manage low-latency multimodal streams and the power of "Human-in-the-Loop" AI. Shifting from static prompts to an active, interruptible conversation with Gemini Live changed how I view software architecture.

What's next for OmniConstruct: Vision-to-Live Architect Moving beyond web apps to full-stack mobile development and integrating multi-user collaborative sketching, where a whole team can draw together while Gemini builds the backend in sync.