-
-
Landing page of our Browser
Asteroids: Navigating the Programmable Web with an AI Agentic Browser
"The web was built for eyes, not for code. We’re changing that."
Inspiration
The modern internet is a massive library designed for human consumption, but for AI agents, it is a "read-only" maze. While APIs exist for some services, the vast majority of the web—legacy enterprise portals, complex Single Page Applications (SPAs), and local government sites—remains inaccessible to autonomous agents. We observed that traditional automation is brittle; the moment a developer changes a single CSS class or an ID, the bot breaks.
We were inspired to build Asteroids to bridge this Action Gap. As students at MSRIT and AWS Community Builders, we wanted to create a "Digital Employee" that doesn't just read the web but navigates and reasons through it exactly like a human would—using sight, intuition, and memory rather than just lines of static code.
What it does
Asteroids is an AI Agentic Browser that acts as a Universal Adapter for the programmable web. Powered by Amazon Bedrock (Nova) and the AWS Strands SDK, it allows agents to perform tasks that were previously impossible without manual human intervention.
Key Features and Capabilities
Multimodal Visual Reasoning & Grounding Asteroids doesn't just parse HTML; it "sees" the layout. By utilizing Nova Pro's multimodal capabilities, the agent understands spatial relationships. If a code-level selector fails, Asteroids takes a screenshot, identifies the target element visually, and retries the action using precise pixel coordinates. This "Self-Healing" logic ensures that UIs which are hostile to standard scrapers are easily navigated.
Deep Memory & Cognitive History (mem0 Integration) Every user has unique preferences. By integrating mem0 and Amazon DynamoDB, Asteroids maintains a persistent memory of user habits. It remembers that you prefer specific filters on search results or that you frequently look up resources in a particular AWS region. It summarizes your browsing history to provide context-aware assistance, allowing you to ask questions like "Find that research paper I was looking at last Tuesday about transformer architectures."
Safety-First Supervisor Mode Autonomy requires trust. We implemented a native safety gate called Supervisor Mode. For high-stakes actions—such as processing a payment, deleting an S3 bucket, or submitting a form—the agent automatically pauses and emits an event via AWS EventBridge. The user receives a notification in the UI and must provide a manual "thumbs up" before the agent proceeds.
Instant Tab Synthesis & Context Merging Asteroids can look across multiple open tabs simultaneously to synthesize information. If you are researching products across three different e-commerce sites, the agent can merge the data, compare specs, and generate a unified report in seconds.
Universal Document Intelligence The browser features a built-in multimodal PDF and document reader. You can fire a task to "Analyze the financial charts in the open PDF tab and compare them with the data on the current website." The agent handles the cross-reference reasoning natively.
How we built it
Building a production-grade AI browser as students required a masterclass in AWS-native optimization. We moved away from simple "automation scripts" toward a robust, event-driven distributed architecture.
- Orchestration via AWS Strands SDK
Instead of writing fragile manual loops, we utilized the AWS Strands Agents SDK. We implemented a custom InjectDOMStateHook (a BeforeModelHook) that automatically captures the browser state before every Bedrock inference call. This ensures the model always has the most current view of the page without manual boilerplate. The agent follows a cyclic reasoning path:
$$\text{Observe (DOM/Vision)} \rightarrow \text{Reason (Bedrock)} \rightarrow \text{Act (CDP)} \rightarrow \text{Validate}$$
- High-Performance "Sidecar" Networking
In cloud environments, latency is the enemy of automation. We deployed our Python agent logic and a headless Chromium instance as sidecars within a single Amazon ECS Fargate task. We used the Chrome DevTools Protocol (CDP) over a local WebSocket (ws://localhost:9222). This kept networking latency to $\Delta t < 2ms$ and avoided the data transfer costs associated with cross-service traffic.
- Intelligent DOM Pre-processing & Pruning
Sending a raw HTML DOM tree (often $>500$ KB) to an LLM is a "token nightmare." We engineered a Pruning Engine that strips non-interactive nodes, removes empty containers, and merges the Accessibility Tree with the standard DOM. This reduces the input context size by $\approx 85\%$, allowing us to stay within the AWS Free Tier while handling complex SPAs.
- Tiered Reasoning Stack on Amazon Bedrock
We optimized our model selection to balance precision and cost. Nova Micro handles 80% of routine text parsing. Nova Lite is used for structured data extraction and "Judge" validation. Nova Pro is reserved for Conditional Vision—it is only triggered when the agent needs to identify elements via screenshots.
Challenges we ran into
Engineering a browser that "thinks" presented hurdles that local scripts never face:
CDP WebSocket Reliability: Silent socket drops on AWS were frequent. We solved this by building a custom retry wrapper with idempotency checks to ensure a click is never fired twice but is always fired exactly once.
The "Cold Start" Problem: Pulling a heavy Chromium image can take significant time. We engineered a "Warm Pool" logic using ECS service auto-scaling to ensure the first user always hits an active, ready-to-browse container.
Spatial Mapping Accuracy: Mapping a visual button $[i]$ in a screenshot to a code-level click(index) requires high precision. We overcame this with heavy few-shot prompt engineering, achieving $99\%$ accuracy in UI element identification.
Extra: Redefining Web Interaction
Traditional tools like Selenium or Puppeteer require days of manual coding for every website and break the moment the site updates. Generic AI agents are often "Read-Only," providing information but unable to act on it.
Asteroids represents a new category: the Agentic Operating System. It provides zero-setup automation that works out of the box on any site, uses multimodal vision to handle UI updates without maintenance, and runs entirely on a scalable, serverless AWS infrastructure.
Accomplishments that we're proud of
Free Tier Supremacy: We successfully architected a system that provides enterprise-level automation while staying strictly within the AWS Free Tier limits.
Visual Grounding Engine: Developing the logic that allows an agent to "heal" itself using screenshots is the crown jewel of our project.
Human-in-the-Loop Integration: Routing safety gates through AWS EventBridge to a custom WebSocket UI ensures that Asteroids is as safe as it is powerful.
What we learned
This project was a deep dive into Distributed Systems and Multimodal AI. We learned that orchestration is just as important as the model itself. Moving from manual code to the AWS Strands SDK taught us the value of "framework-first" scaling. We also realized that even as students, we can build world-class infrastructure by leveraging the depth of the AWS ecosystem.
What's next for Asteroids
OpenClaw Integration: Allowing users to fire off complex research tasks from their mobile phones via OpenClaw and find the results waiting on their desktop.
Agentic Skill Marketplace: A platform where users can share "Asteroid Skills"—specialized toolsets for navigating specific complex enterprise UIs.
Autonomous Auth Handshake: Developing a "Local Vault" that can securely handle 2FA prompts via a mobile-to-browser handshake using encrypted SQS cues.
This article and the associated Asteroids architecture were co-authored by our Asteroids Agents.
Log in or sign up for Devpost to join the conversation.