Inspiration
"It works on my machine."
Every developer has said it. Every QA engineer has hated hearing it. The disconnect between a user reporting a bug and a developer trying to reproduce it is the single biggest time-sink in software engineering. We realized that traditional bug reports are missing two critical dimensions: Context and Emotion.
A Jira ticket saying "Button doesn't work" is useless. But a video of a user clicking that button 50 times while shouting in frustration? That tells a story of a critical failure.
We built Auto-QA to bridge this gap. We wanted to create an agent that doesn't just "log" bugs, but witnesses them, understands the user's pain, and instantaneously writes the code to fix it.
What it does
Auto-QA is a multimodal agent that acts as a Senior SDET (Software Development Engineer in Test) and a User Researcher simultaneously.
Multimodal Bug Capture: The user simply records their screen and voice while reproducing a bug.
"Rage" Detection: Using Gemini 3's native multimodal capabilities, the agent analyzes audio tonality and mouse velocity to calculate a "Frustration Index."
Auto-Reproduction: It parses the video frame-by-frame to generate a robust Playwright (TypeScript) test script that replicates the exact user flow.
Self-Healing Code: It infers the underlying tech stack (e.g., React, Vue) from the visual behavior and generates a candidate Code Fix to patch the issue.
How we built it
We built the entire application using Google AI Studio's "Build" mode, leveraging the Gemini 3 Pro model.The core logic relies on a custom "Severity Algorithm" that fuses technical severity with user sentiment. We modelled the priority score $P$ as a weighted function of technical failure $T$ and emotional intensity $$P = \frac{1}{1 + e^{-(w_t T + w_e E + b)}}$$ Where:$T$ is the technical severity (crash vs. cosmetic), inferred from visual error toasts.$E$ is the "Rage Score," derived from audio amplitude and rapid-click frequency.$w$ are the learned weights from our system prompt. We used Gemini 3's 2M+ token context window to hold the entire video buffer in memory, allowing the model to correlate a specific "grunt" of frustration at timestamp 0:14 with a specific UI freeze at 0:14.
Challenges we ran into
Latency vs. Accuracy: Initially, processing high-res video took too long. We optimized this by downsampling the video before sending it to the API, relying on Gemini's ability to "reason" from lower-fidelity inputs.
The "Sarcasm" Problem: Early versions of the agent thought users laughing at a bug were "happy." We had to tune the system prompt to recognize "sarcastic laughter" as a sign of high frustration.
Vibe Coding: Building the UI entirely through prompts in AI Studio was a new experience. It required us to "program in English," iterating on the visual design (the Cyberpunk aesthetic) by describing feelings rather than writing CSS.
Accomplishments that we're proud of
The "Rage Click" Detector: It genuinely works. If you shake your mouse angrily, the dashboard lights up red. It feels magical.
Zero-Shot Code Generation: The agent successfully wrote a working Playwright script for a complex scroll-virtualization bug without ever seeing the source code, just by watching the video.
The UI: We went from a blank page to a fully functional, professional-grade "Command Center" interface in under 3 hours using Gemini 3's coding capabilities.
What we learned
"Vibe Coding" is a real paradigm shift We went into this expecting to write a lot of React glue code. Instead, we spent 80% of our time "vibing" with the model in AI Studio. We learned that with Gemini 3, you don't just prompt for function ("make a button"); you prompt for feeling ("make it look like a cyberpunk terminal that warns you of danger"). The model's ability to translate abstract aesthetic concepts into concrete CSS (e.g., the neon glow effects on the high-severity alerts) was a massive productivity multiplier.
Context Engineering > Prompt Engineering With Gemini 3's 2M+ token context window, we learned that the "old way" of RAG (retrieving tiny chunks of data) is obsolete for this use case. We could feed the entire video buffer and the entire page source into the model at once. This allowed the agent to find non-obvious correlations—like noticing that a specific CSS animation class on line 402 was triggered exactly when the user screamed in the video. We learned that giving the model "the whole picture" yields significantly smarter reasoning than trying to curate inputs.
Multimodality requires new "Safety Rails" We discovered an interesting edge case: "Sarcastic Success." Early testing showed that when a user laughed sarcastically at a bug, the model initially tagged it as "Positive Sentiment." We learned we had to explicitly instruct the model on the nuances of developer frustration (e.g., “If the user laughs while clicking rapidly, this is rage, not joy”). This taught us that multimodal agents need distinct behavioral tuning compared to text-only bots.
The "Agentic Gap" is closing Previously, getting an LLM to write a complex Playwright script required 3-4 turns of "back and forth" correction. Gemini 3’s reasoning capabilities allowed it to "simulate" the run in its head. We learned that the model can effectively "self-correct" its code before outputting it, drastically reducing the need for a human-in-the-loop to fix syntax errors.
What's next for Auto-QA: The Rage Agent
Jira/Linear Integration: Automatically posting the bug report to the team's tracking board.
Live "Co-Pilot" Mode: Running in the background and automatically capturing the last 30 seconds when it detects a "Rage Click" (similar to a dashcam for coding).
Enterprise Context: Feeding the agent the actual GitHub repository so the "Suggested Fix" is not just a snippet, but a Pull Request ready to merge.
Built With
- css3
- gemini
- genai
- google-ai-studio
- html5
- javascript
- multimodal
- playwright
- react
- screen-capture-api
- typescript
- web-speech-api
