GIF
open comet demo
Website

🧠 Open Comet

Inspiration

Modern AI agents can generate text—but they struggle with real-world execution on the web. Tasks like research, form filling, or multi-step workflows often break due to dynamic UIs, unreliable selectors, and lack of structured planning.

We were inspired to solve a core gap:

How can we build an AI agent that doesn’t just respond—but plans, verifies, and reliably executes tasks inside a real browser?

This led to Open Comet: a spec-driven autonomous browser agent designed for reliability, transparency, and privacy.

What it does

Open Comet is an AI-powered Chrome extension that turns high-level user intent into structured, executable web workflows.

It can:

Perform multi-step browsing tasks across websites
Run deep research loops across tabs and sources
Interact with web pages using visual + DOM understanding
Recover from failures using self-healing execution strategies

Unlike traditional agents, Open Comet:

Plans → verifies → executes instead of blindly acting
Uses multimodal grounding (DOM + screenshots)
Maintains a Zero-Data architecture (all sensitive data stays local)

How we built it

We followed a spec-driven development approach, where every task is converted into a structured execution pipeline before action.

🔧 Core Architecture

Intent → Spec Layer Converts user prompts into structured task graphs
Execution Engine Breaks tasks into atomic browser actions (click, type, navigate)
Multimodal Perception Layer Combines:
- DOM parsing
- Screenshot-based visual grounding
Verification Loop Confirms each step before proceeding (reduces hallucinated actions)
Self-Healing System Handles:
- DOM changes
- stale selectors
- dynamic layouts
Model Abstraction Layer Supports multiple providers:
- OpenAI
- Anthropic
- Google
- Mistral AI
- Ollama
Privacy Layer (Zero-Data Architecture) Ensures browsing data and history remain on-device

Challenges we ran into

Dynamic Web Environments Websites frequently change structure, breaking naive automation.
Reliable Element Detection Pure DOM-based approaches failed—required hybrid visual + structural grounding.
Agent Consistency Early prompt-based agents behaved unpredictably without structured specs.
Latency vs Accuracy Tradeoff Adding verification improved reliability but increased execution time.

Accomplishments that we're proud of

✅ Built a fully working autonomous browser agent
✅ Implemented spec-driven execution (not just prompt-based AI)
✅ Designed a self-healing interaction system
✅ Achieved high task success rate across dynamic websites
✅ Created a privacy-first Zero-Data architecture
✅ Enabled multi-model support (cloud + local LLMs)

What we learned

Spec-driven > prompt-driven for reliable AI systems
Multimodal grounding is essential for real-world execution
Verification loops drastically improve agent stability
Real-world environments expose edge cases not seen in controlled demos

This project shifted our mindset from:

“Ask AI to do things” → “Design systems that ensure AI does things correctly.”

What's next for Open Comet

🔄 Advanced Planning Engine Hierarchical task decomposition and long-horizon workflows
🧠 Learning-based Skill System Agents that improve from past executions
🔍 Better Execution Trace Visualization Debugging and transparency tools for developers
🌐 Cross-browser + OS-level automation Expanding beyond Chrome
🔐 Stronger Security Layer Sandboxed execution and permission-aware actions