Clark — General-Purpose Autonomous Web Agent

Inspiration

We were tired of doing the same boring browser work over and over — and frustrated that existing tools either needed hard-coded scripts or could only talk about the web without actually doing anything on it. OpenAI Build Week gave us the push to build something better.

What it does

Clark is an autonomous web agent. Tell it what you need in plain language, and it opens a real browser and gets it done — searching, filling forms, logging in, downloading files — while pausing for your approval on anything sensitive. It works on any website, not just ones it was pre-programmed for.

How we built it

A Python / FastAPI backend runs a ReAct reasoning loop against a real Chromium browser via Playwright — no headless scraper, a real visible window the user can watch and interact with. The AI inference runs on Fireworks AI (serving models like GPT-OSS-120b and DeepSeek-V4-Pro) with Google Gemma 4 as an automatic fallback. A Next.js 14 frontend streams every thought, tool call, and observation live over Server-Sent Events. Profile, credentials, and payment data are stored as local JSON — no external database. The project was developed end-to-end with OpenAI Codex.

Core architecture

User → Next.js UI → FastAPI API → AgentRun (ReAct loop) → LLM (Fireworks / Gemini)
                                                          → Tools (~20) + real browser
                                  ← SSE stream (live trace)

Challenges we ran into

• Grounding clicks without brittle selectors — solved with a numbered Set-of-Marks overlay that labels every interactive element on the page, so the agent clicks "box #14" instead of a CSS/XPath selector a redesign can silently break.
• Keeping the agent honest about what it actually did — solved by re-deriving final answers from recorded tool-call arguments instead of the model's memory of them. The gap between what a model thinks it did and what it actually did is real and measurable.
• Preventing the model from ever seeing passwords or OTPs — solved by making that structural: the tool layer has no code path to send sensitive values to the AI. Credentials, one-time codes, and CAPTCHAs are injected directly by the backend after the human provides them.
• Fitting long multi-step tasks into limited context windows — solved with automatic message compaction and a capped step budget that forces a "wrap up now" instruction near the limit, so tasks never end in silence.
• Surviving anti-bot defenses — persistent Chrome profiles, stealth patches for common headless tells, and human-like typing/mouse movement improve pass rates on real government and enterprise portals.

Key features

• ~20 tools — web search, arXiv search, browser navigation, Set-of-Marks clicking, smart text/date filling, auto-login, OTP/ CAPTCHA handling, document generation, and more
• Adaptive form filling — identifies form fields by meaning, fills from a saved profile, and asks only for what's missing
• Live streaming trace — every thought, action, and observation streams in real time, replayable later from History
• Human-in-the-loop modals — dedicated masked forms for credentials, OTP codes, CAPTCHAs, and payment review
• Grounded final answers — summaries are re-derived from actual tool arguments, not the model's potentially hallucinated narrative
• Page state classification — detects blocked, truncated, paywalled, error, and login pages so the agent reacts correctly instead of confidently reporting "nothing found"
• Full audit trail — every conversation is persisted as a replayable, timestamped transcript

Technology stack

What we learned

Building an agent is fundamentally different from building a chatbot — every design decision has to account for real-world consequences. Human-in-the-loop has to be architectural, not advisory. The model's ability to narrate plausible-sounding actions it never actually took means grounded outputs aren't optional — they're the only way to trust what the agent reports.

What's next for Clark

• Near-term: Unifying CLI and web execution paths, adding comprehensive PII redaction, expanding MFA support, and building structured reliability benchmarks
• Longer-term: Support for more LLM providers (Claude, local models via Ollama), team features with shared credential vaults, and a companion mobile experience for approving human-in-the-loop gates on the go

Built for OpenAI Build Week. Developed with OpenAI Codex.

Built With

• ai
• api
• chatgpt
• fastapi
• fireworks
• gpt
• nextjs
• tech