Astra

• 

  This is the architecture diagram of Astra

Inspiration

I was inspired by the movie Iron Man, where Tony Stark has his personal assistant, JARVIS, that doesn't just answer questions, but actively assists him in his tasks and physically performs them like a partner. I wanted to bring that level of seamless, voice-driven autonomy to the web to life.

What it does

Astra is an autonomous, voice-controlled web co-pilot. Instead of just summarizing text or acting as a chat window, Astra physically drives your Chrome browser for you. It looks at the screen, understands the UI, and uses mouse and keyboard inputs to navigate complex web apps, fill out forms, and complete tasks while you sit back.

How we built it

I've built a hybrid architecture that separates the AI's "Brain", "Ears", and "Hands".

The Ears (Frontend): I have built a lightweight Chrome Extension using Manifest V3. It captures 16kHz PCM audio via an invisible offscreen document and renders a reactive, 3D Siri-like orb using the Web Audio API.

The Brain (Middleware): A local Python FastAPI server uses WebSockets to stream the audio directly into Google Gemini Multimodal Live API.

The Hands (Backend):I've used Playwright connected to a live Chrome session via a Chrome DevTools Protocol backdoor (localhost:9222). Gemini calls Python tools to run an analysis on the screen, which temporarily draws numbered bounding boxes over clickable elements, takes a screenshot, and sends it back to the model. Gemini then executes hardware-level clicks and typing based on those visual anchors(due to the restrictions of Chrome security).

Challenges we ran into

Honestly, making this work natively was a massive headache.

The Anti-Bot: Modern React sites(for this case is trip.com) actively block synthetic JavaScript clicks to prevent botting. I had to bypass this entirely by forcing Playwright to simulate actual human reaction delays and raw hardware clicks.

Chrome's Strict Security: Getting a Chrome Extension to stream background microphone audio without throwing errors or violating Content Security Policies took hours of debugging.

The Infinite Loop: Initially, if the agent clicked the wrong thing, it would panic and spam the same tool call forever. I had to add a strict "Anti-Loop Protocol" into the agent's configuration and write rich error handling so the AI knew when to stop and ask the human for help.

Accomplishments that we're proud of

I am proud of the fact that I managed to get one of the hardest features to work which was the Interruption feature where the user can interrupt the agent by giving a new command and the agent would pivot over to the new task without hallucinating.

What we learned

I've learnt a massive amount about asynchronous Python , WebSocket streaming, the architecture of Chrome Extension . More importantly, I've learnt how to ground an LLM. By feeding Gemini annotated screenshots instead of massive, HTML DOM trees which are messy and unstructured, we learned how to make AI see the web exactly like a human does.

What's next for Astra

Right now, Astra requires a local Python backend to execute the CDP hardware clicks. The next step is exploring ways to package this into a native desktop application (like Electron) so anyone can install it into their desktop. so it can securely remember your shipping addresses, preferences, and payment details to automate tedious tasks across any website you visit.

Built With

• chrome
• css
• fastapi
• html
• javascript
• playwright
• python
• uvicorn
• vertex-ai
• websockets