Glass

• 

  Local Git safety and coding agents with ADK and Gemini planning on Cloud Run.

Inspiration

When a visual bug becomes a ticket, the reviewer often has to reconstruct what they just saw. The coding agent gets a screenshot and a sentence, but no reliable path to the responsible source. We built Glass to keep the selected element and its browser context attached to the instruction.

What it does

Glass adds an annotation and voice layer to a running web application. The user selects the broken element, saves an annotation, and speaks the desired correction. Glass captures the transcript together with the element selector, geometry, nearby text, accessibility information, computed styles, React component ancestry, and a source-file hint.

The local daemon creates a Git checkpoint and launches an AGY coding agent against the selected repository. Job phases stream to the browser over server-sent events while the user continues reviewing the page. When the edit lands, Vite updates the interface in place. Glass reports the changed files and verification result, and the checkpoint can restore the repository if the result is wrong.

For complex instructions, the daemon can call a planner deployed on Google Cloud Run. That service uses Google ADK and Gemini 3.5 Flash through Vertex AI to turn the complaint and browser evidence into an implementation plan. The cloud service never receives repository contents or write access; the local agent remains the only component allowed to edit code.

How we built it

The browser surface is a React and TypeScript annotation toolbar. A local Node.js daemon owns job scheduling, Git safety, transcription, verification, file watching, and the SSE event stream. AGY performs the background code edit. The hosted planner is a separate Cloud Run service built with Google ADK and Gemini 3.5 Flash on Vertex AI.

Google Cloud handles intent planning while source code and credentials stay on the developer's machine. The Cloud Run planner returns a plan; it cannot execute an edit. The daemon validates the request, checkpoints the repository, runs one job at a time, and verifies the changed files. Its revert operation is idempotent.

Challenges

The hardest part was deciding when an edit had succeeded. A coding process can exit successfully without changing a file, and a repository can have failing checks before the agent touches it. Glass checks Git state rather than trusting the agent's status flag. It compares verification failures with the changed-file set to distinguish a verified change from a proven regression or a change that could not be checked.

Voice input needed a way to catch mistakes before sending. Glass supports push-to-talk with Alt+Space and an explicit Speak button. It shows when recording is active, lets the user review the transcript after stopping, and sends only after they confirm the text.

Accomplishments

Glass takes a visual selection through to a reversible repository edit. It includes a background queue, persisted job events, authenticated mutation routes, loopback-only CORS, source-context capture, speech fallback, live page updates, verification, and safe revert behavior. The automated suite covers 363 unit and integration tests plus a 24-step end-to-end run that starts a real daemon, opens the built demo, creates an annotation, launches a deterministic test agent, observes SSE events, and confirms the fixture changed.

What we learned

We spent as much effort on the execution rules as on the model: preserving context, showing progress, controlling repository access, and reporting when no reliable verification ran. Separating cloud planning from local execution gave each component a clear responsibility and kept the deployment smaller.

What's next

The next step is visual verification against the original selected region, followed by durable queue storage for jobs that should survive a daemon restart. The same context pipeline can also support design-system migrations and accessibility remediation across multiple selected elements.

Technologies

Gemini 3.5 Flash, Vertex AI, Google ADK, Google Cloud Run, TypeScript, React, Node.js, Vite, AGY, SQLite, Server-Sent Events, Git, Vitest, and ElevenLabs.

Pre-existing code disclosure

The annotation toolbar and MCP baseline are derived from Agentation by Benji Taylor and retain their PolyForm Shield 1.0.0 license. Glass adds the voice workflow, local daemon, background-agent execution, Git safety, verification, demo, and Cloud Run planner. NOTICE and PROVENANCE.md document the attribution and the boundary between original and derived code.

Built With

• gemini
• typescript