Oye

Oye is an AI teammate that can understand your request, operate the browser for you, show you what it’s doing live, and let you step in instantly when human judgment is needed.

Comment

Inspiration

We wanted to build an AI system that does more than answer questions. Most assistants are still trapped in a chat box: they can explain, summarize, and generate, but they cannot reliably act with you, show you what they are doing, or hand control back when a human needs to step in. Real work often lives inside browsers, forms, dashboards, booking flows, admin tools, and sign-in walls. That gap between “AI can suggest” and “AI can actually help me finish the task” is what inspired Oi.

Our goal was to build an AI operator that feels collaborative rather than magical: you can talk to it with text or voice, let it navigate and reason through tasks, watch a live browser view, and immediately take over when human judgment is required.

What we built

Oi is a multimodal AI operator across web, desktop, and mobile. It combines:

  • conversational AI
  • live browser automation
  • real-time browser frame streaming
  • human-in-the-loop takeover
  • local and remote runner support

In practice, that means a user can ask Oi to do something in natural language, Oi can reason through the request, operate a browser, stream what it sees back into the product, and let the user step in at any point.

The system is split into several parts:

  • a FastAPI backend for orchestration, auth, sessions, browser control, and API surfaces
  • an automation runtime for long-running execution
  • a web app for chat, sessions, and live browser viewing
  • an Electron desktop app that can register a local runner and connect to a real local browser
  • a mobile app for multimodal access
  • Gemini model integrations for planning, reasoning, and live multimodal interaction

How we built it

We designed Oi as a multi-surface system rather than a single demo page.

The backend is the coordination layer. It handles authenticated sessions, automation routes, browser session state, live streaming endpoints, and communication with runners. We used FastAPI because we needed a clean API layer, strong typing, and good support for both HTTP and realtime flows.

The automation runtime is separated from the backend so orchestration and execution are not tightly coupled. That made the system easier to reason about and much closer to a production-style architecture, where task execution, browser control, and request handling are different responsibilities.

For the product surfaces, we built:

  • a Vite + React web app as the main user interface
  • an Electron desktop app for local runner and browser control flows
  • a React Native mobile app for multimodal and cross-device access

For browser automation, we supported both:

  • a local runner, where the user’s own desktop/browser participates
  • a remote runner, where execution can happen remotely

One of the most important parts of the project was the live session transport. We built the flow so that the runner can capture browser state and screenshots, send heartbeats and frames to the backend, and the UI can render that live stream using WebSocket or SSE fallbacks. That turned the product from a black-box agent into something inspectable and collaborative.

On the AI side, we integrated Gemini in two ways:

  • standard Gemini model calls for planning and reasoning
  • Gemini Live for realtime multimodal interactions

What we learned

The biggest lesson was that building a useful AI product is much more about systems design than prompt design.

It is not enough to connect a model to a frontend. Once an agent starts acting in the real world, you need:

  • state management
  • execution/runtime separation
  • realtime transport
  • observability
  • fallback paths
  • human control boundaries

We also learned how important it is to design for trust. Users are much more comfortable with AI actions when they can:

  • see the browser live
  • understand what step is happening
  • interrupt or take control
  • recover from errors without starting over

From an engineering perspective, we learned a lot about:

  • structuring browser-session ownership and control
  • handling realtime frame streams robustly
  • connecting local desktop/browser execution with cloud orchestration
  • keeping a multi-app monorepo coherent through shared types and common contracts

Challenges we faced

The hardest part was not generating responses. It was getting the whole system to behave like a single product.

Some of the key challenges were:

  • keeping browser session state consistent across backend, runner, and UI
  • handling websocket and SSE edge cases for live browser frames
  • making local runner and live browser flows stable enough for real demos
  • debugging auth/session issues across web, desktop, and backend boundaries
  • making realtime multimodal behavior work without turning the system into a fragile one-off demo

A recurring challenge was payload consistency across transports. For example, live frame streaming involved multiple paths: direct frame fetches, SSE streams, and websocket streams. Small shape mismatches in those payloads could cause the UI to show metadata without rendering the actual frame. Fixing those issues taught us how important strong cross-surface contracts are in a multi-runtime system.

We also had to think carefully about scaling behavior and ownership semantics. A browser session is not just data; it is a live control surface. That means session routing, runner identity, and handoff rules matter a lot.

Why this project matters to us

Oi is our attempt to make AI feel genuinely useful in the loop of real digital work. Instead of being another assistant that stops at advice, Oi tries to become an operator that can reason, act, show its work, and collaborate with a human in real time.

That combination of multimodal interaction, browser execution, live visibility, and human takeover is what makes the project exciting for us. It points toward AI systems that are not just smart, but actually usable.

Built With

Share this project:

Updates

Submission history