ORION

Inspiration

People spend a huge amount of time repeating the same work online: searching websites, applying filters, opening results, comparing information, and verifying whether everything actually matches their requirements.

Browser agents can help, but they often approach every request as a completely new problem. Even after successfully navigating a website once, an agent may need to rediscover the same process during the next run.

We asked ourselves:

What if you could describe your goal once and let an organized team of agents plan, execute, verify, and remember the workflow?

That idea became ORION.

What it does

ORION is a multi-agent infrastructure for completing web tasks from a natural-language request.

At its core are two systems: ARGUS, which controls the work, and Ghost API, which remembers successful procedures.

ARGUS interprets the user’s request, checks whether it is supported and safe, and turns it into a bounded execution plan. It assigns focused subtasks to browser workers, coordinates their dependencies and limits, and manages the run from the initial request to the final answer.

The browser workers interact with websites through Steel and return structured reports containing results, actions, evidence, checks, and failures. A moderator evaluates and combines those reports, while ARGUS validates the selected information before presenting it to the user.

Ghost API gives ORION procedural memory. It records successful browser workflows, tests whether they still work with different inputs, and makes qualified procedures available for future requests.

ARGUS decides and coordinates what needs to happen. Ghost remembers how to do it again.

How we built it

The backend is written in Python, while Mission Control uses browser-native HTML, CSS, and JavaScript without a frontend framework or build process. We use FastAPI for the APIs that power Mission Control and Ghost, including request submission, live run updates, workflow lookup, and run history.

From natural language to a bounded plan. When a user submits a request, ARGUS converts it into structured goals, parameters, constraints, and success criteria. It checks whether the request is clear, supported, and safe before any browser work begins. ARGUS then creates only the subtasks needed for that request and gives each one an expected output and an execution budget.

The plan can include independent tasks or tasks that depend on earlier results. ARGUS controls their order, tracks their progress, and prevents the workflow from growing into an unlimited agent loop.

From a plan to real browser actions. ARGUS sends each subtask to a specialized browser worker. Our DOM worker creates a browser session through Steel and uses Playwright to inspect and control the page. It can find controls, enter values, apply filters, navigate, and extract current information.

The worker follows an observe–act–verify loop. It examines the current page, chooses a permitted browser action, performs it, and observes the page again to confirm what changed. It also checks details such as applied filters, result fields, prices, source URLs, and empty-result states instead of assuming that a successful click means the task is complete.

We also built a separate visual worker using UI-TARS for pages that cannot be understood reliably through their DOM. The system can prepare a handoff containing the task, browser context, screenshot, and remaining budget. The visual worker itself is implemented, while the complete live recovery handoff is still being integrated.

From worker reports to a trusted result. Every worker returns a structured report rather than an unrestricted answer. These reports include the information found, the actions taken, supporting evidence, validation checks, and any failures.

The moderator evaluates the reports, removes duplication, and selects the records relevant to the request. It cannot freely introduce new facts into the final response. ARGUS checks that the selected information came from the current run and builds the user-facing answer from those verified fields.

From successful execution to reusable memory. Before exploring a website, the browser worker asks Ghost API whether a compatible qualified procedure already exists.

If Ghost finds one, the worker binds the user’s new inputs to the saved steps and replays the procedure against the current page. It does not reuse an old answer—the website is visited again, fresh information is extracted, and the result is validated again.

If no compatible procedure exists, the worker explores the website normally. A successful and verified trace can then be stored as a candidate workflow. Ghost tests that candidate through three additional runs with changed inputs, including a case where no results should be returned. Only after those checks pass is the workflow qualified for automatic reuse.

Making the infrastructure visible. As the request moves through interpretation, planning, browser execution, moderation, validation, and Ghost, the backend streams events to Mission Control. The dashboard shows the execution plan, active workers, browser actions, Ghost decisions, validation checks, failures, and final result.

We verified this connected workflow through Steel. ARGUS interpreted and planned the request, the DOM worker explored the hosted catalog, Ghost saved and qualified the workflow, and a later Mission Control run replayed it with new inputs and zero browser-worker model calls.

Challenges we ran into

Connecting independently built components. ARGUS, Ghost, the moderator, the DOM worker, the visual worker, and Mission Control were developed as separate modules. We had to define shared requests, reports, evidence, and failure states so the pieces could communicate without creating conflicting controllers.

Knowing when a task is actually complete. A browser agent can navigate correctly and still return an incorrect or incomplete answer. We separated browser execution, worker completion, moderation, validation, and final publication so each stage could be checked independently.

Knowing when a workflow is reusable. One successful run does not prove that a procedure will work again. This led us to separate new workflows into candidate and qualified states and require additional runs with changed inputs before reuse.

Managing real browser sessions. Working through Steel required careful session creation, cleanup, timing, and error handling. We added execution budgets, fresh page observations, bounded retries, and explicit failures to keep browser problems visible.

Handling different kinds of websites. Some websites expose useful structure through the DOM, while visual or canvas-based interfaces may not. This pushed us to develop both DOM-based and vision-based browser workers.

Accomplishments that we're proud of

Building the infrastructure. Instead of connecting a single AI call to a frontend, we created separate layers for interpretation, planning, browser execution, moderation, validation, memory, and monitoring.

Connecting the core workflow. We ran a natural-language request through Mission Control, the real ARGUS interpreter and planner, a Steel-powered DOM worker, Ghost API, the moderator, validation, and the final controller-generated answer.

Qualifying reusable workflows. We demonstrated Ghost’s full initial lifecycle: exploration, candidate creation, three changed-input qualification runs, and later reuse with fresh results.

Replaying with zero browser-worker model calls. Once the workflow was qualified, Ghost replayed it through Steel without requiring the browser worker’s model to rediscover the steps.

Keeping final claims tied to evidence. The moderator selects observed records rather than writing an unrestricted answer. ARGUS validates those selections and constructs the final response from information produced during the current run.

Together, the components form a digital assembly line for web tasks:

ARGUS is the control system.
Ghost API is the memory system.
Subagents are the execution layer.
The moderator provides quality control.
Mission Control makes the process visible.

What we learned

We learned that reliable agents depend on much more than choosing a powerful model.

Infrastructure matters. Clear responsibilities, bounded plans, compatible interfaces, browser-session management, and honest failure handling are essential when several intelligent components work together.

Evidence matters. Completing a browser action is not the same as completing the user’s request. Results need to be checked against current page observations and the original success criteria.

Memory needs qualification. Saving browser actions is easy. Proving that those actions still work with new inputs—and still produce correct results—is much harder.

Most importantly, reliable agents need planning, execution, verification, coordination, and memory.

What's next for ORION

Our next step is to expand the verified workflow beyond the controlled product catalog and support more websites and task types.

We want to expand ORIONS capabilities beyond our saved logs. We want to impliment smart systems to combat antii-scraping website firewalls.

We want to complete the live DOM-to-visual recovery path, improve evidence capture, and add bounded workflow repair when websites change. Repaired procedures would become new candidates and need to qualify again before reuse.

We also want to develop secure support for interactive workflows. Actions such as submitting forms or applications would require explicit user confirmation, a clear preview of what will be sent, and stronger privacy protections.

Our long-term goal is simple:

Give ORION a goal. Let the agents handle the web.