Robin

Computer-Use Google Meet Co-Worker

Comment

Summary

Most meeting assistants document what happened and leave the actual work to people. Robin takes ownership of the work while the meeting is still happening.

Robin joins Google Meet as its own participant. A teammate gives one spoken request—“Robin, compare the quarterly finance files and make a few slides”—and Robin drives the rest of the workflow. It acknowledges the task, chooses relevant approved sources, builds a grounded analysis with GPT-5.6, validates its citations and calculations, generates the deliverables, raises its hand when ready, presents the deck once invited, and immediately returns to listening for follow-ups.

The result feels less like prompting a chatbot and more like delegating an outcome to a coworker who is already in the room.

Inspiration

Meetings constantly create work: compare results, investigate a question, build slides, summarize customer feedback, or prepare a follow-up. Today, that work usually stalls until someone leaves the conversation, locates the right files, performs the analysis, creates an artifact, and schedules another review.

We wanted an AI that could close that loop before the meeting ended. Instead of opening another chat window and manually guiding every step, teams should be able to delegate an outcome once. The AI should stay attentive, recognize when it is addressed, work independently on its own machine, show its sources, signal when it is ready, and explain the result to everyone.

That idea became Robin.

What it does

After the operator launches Robin and supplies a Meet link, Robin runs the work loop end to end:

  1. The operator starts Robin with one command: make robin.
  2. Robin launches a dedicated Chrome profile, FastAPI runtime, Next.js dashboard, and native macOS bridge.
  3. The operator pastes a Google Meet URL and selects Join & listen.
  4. Robin joins muted, monitors the conversation, and recognizes complete turns addressed to “Robin.”
  5. A participant delegates an outcome in natural speech.
  6. Robin acknowledges it aloud and starts working without additional operator input.
  7. GPT-5.6 selects and inspects relevant files through bounded workspace tools, then produces a cited deliverable.
  8. Robin generates a browser deck, PowerPoint export, Markdown report, and validation evidence.
  9. Robin checks its own structure, calculations, citations, and source lineage.
  10. As soon as the result is verified, Robin raises its hand to signal that it is ready.
  11. Once invited, Robin completes the Meet screen-share flow, narrates the deck, and advances every slide.
  12. Robin stops sharing cleanly and resumes listening for grounded questions or revisions.

Robin also drives its own recovery: it can reconnect to an existing Meet tab, wait for admission, fall back to live captions when native audio is silent, suppress its own echoed speech, retry recoverable failures, and clear stale presentation state. The dashboard exposes every decision and transition in real time.

How we built it

Robin combines a local web stack with native macOS capabilities:

  • FastAPI and Python form the control plane for meeting state, transcripts, tasks, agents, artifacts, speech, presentation state, persistence, and observability.
  • Next.js and TypeScript power the operator dashboard and the browser-rendered presentation surface.
  • GPT-5.6 is the primary grounded task-generation model. It plans the work, decides which approved sources matter, calls the required workspace tools, and submits a structured deliverable.
  • OpenAI speech models provide Robin's spoken acknowledgements and slide narration.
  • OpenAI realtime transcription is the primary streaming speech path when native Chrome audio is available.
  • Google Meet captions provide a resilient fallback when macOS ScreenCaptureKit returns silence. Robin stabilizes evolving caption rows, attributes the speaker, ignores prior caption history, and only promotes ordinary turns containing the wake word.
  • Playwright and Chrome DevTools Protocol control the normal Meet interface and recover existing meeting tabs.
  • A Swift macOS bridge handles audio playback, ScreenCaptureKit integration, device inspection, and permission checks.
  • BlackHole 2ch acts as Robin's virtual microphone, allowing generated speech to enter the meeting as real participant audio.
  • Native computer-use automation handles Chrome's screen-share picker because browser automation cannot directly control privileged operating-system dialogs.
  • SQLite and RobinWorkspace preserve transcripts, tasks, artifacts, events, metrics, logs, and rehearsal evidence locally.

How GPT-5.6 is used

GPT-5.6 is not used as a generic text box or a one-shot content generator. It operates inside a bounded agent loop that lets Robin plan and execute grounded knowledge work.

For every accepted request, Robin gives GPT-5.6 the task, relevant meeting context, an allowlisted workspace inventory, and a small set of controlled tools. GPT-5.6 decides which sources to inspect, reads them, synthesizes the evidence, and submits a structured presentation. It cannot browse arbitrary paths, modify original sources, reveal secrets, or cite files it did not read.

The generation prompt asks GPT-5.6 to identify the audience, define the decision or understanding the deck should enable, establish one central takeaway, and build a cumulative story. It uses takeaway-style titles, concise evidence, decision-relevant metrics, and human-readable citations.

Robin does not stop when the model returns text. It validates slide structure, citation coverage, source lineage, output files, and deterministic calculations where applicable. Only after those gates pass does Robin mark the work verified and proactively request the floor.

How Codex accelerated the project

Codex was our primary engineering collaborator throughout Build Week. It helped turn the original product concept into a working cross-stack system spanning Python, TypeScript, Swift, browser automation, audio routing, model tools, and live macOS testing.

Codex accelerated:

  • Translating the product idea into a PRD, technical design, runtime state machine, and implementation plan.
  • Building the FastAPI runtime, Next.js dashboard, Swift bridge, workspace tools, agent loop, and test infrastructure.
  • Designing the grounded GPT-5.6 prompt and structured deliverable contract.
  • Debugging real Google Meet DOM changes, admission flows, captions, mute controls, hand raising, and presentation state.
  • Diagnosing BlackHole routing, speech playback, ScreenCaptureKit silence, and realtime transcription failures from persisted logs and live events.
  • Using browser and computer-use capabilities to rehearse flows that crossed the web/native boundary, especially Chrome's screen-share picker.
  • Creating regression tests after every real-world failure instead of applying one-off demo patches.
  • Improving the slide design and narration workflow after observing real presentations.
  • Reviewing logs, measuring resource usage, maintaining documentation, and preparing the public repository for judges.

The key product decisions remained ours: Robin should be an independent meeting participant; it should require the wake word; source access should be bounded; claims should be cited; it should raise its hand before presenting; and the operator should see every important state transition. Codex dramatically shortened the path from each decision to a tested implementation.

Key technical decisions

A participant, not a hidden bot

Robin uses a dedicated Chrome profile, Google account, microphone route, workspace, and presentation surface. That makes its presence and actions visible to everyone in the meeting.

Own the workflow after delegation

The wake word gives Robin clear authorization to begin. From that point forward, Robin plans the task, chooses sources, creates and validates the deliverables, signals readiness, presents, cleans up, and resumes listening without an operator advancing each stage.

Grounding before autonomy

The agent cannot access the whole computer. It works inside an approved workspace and must read a source before citing it. Generated files are isolated from originals.

Ask for the floor

Robin proactively raises its Meet hand the moment verified work is ready, then waits for an invitation before sharing. It takes initiative without taking control away from the room.

Observable by default

The dashboard streams runtime state and events instead of showing a generic spinner. Operators can see listening, task execution, tool calls, validation, speech, presentation, failures, and recovery as they happen.

Layered recovery

Real meetings are messy. Robin actively monitors its environment and recovers an existing Chrome tab, waits for host admission, switches to captions when native audio is silent, suppresses its own echoed voice, rejects duplicate requests, and cleans up stale presentation sessions.

Challenges we ran into

Capturing real meeting audio on macOS

The hardest problem was not model intelligence—it was audio routing. Chrome, BlackHole, ScreenCaptureKit, Meet's microphone controls, and macOS permissions all interact. ScreenCaptureKit sometimes opened successfully while returning digital silence.

We solved the user-facing failure with a layered approach. Native audio remains the preferred path, but visible Meet captions now serve as a continuous fallback. The caption listener stabilizes partial text, avoids replaying historical captions, tracks speakers, requires the wake word, and suppresses Robin's own speech.

Browser automation versus native dialogs

Playwright can control the Meet page, but Chrome's screen-share picker is a privileged native dialog. We built a hybrid workflow that uses semantic browser automation for Meet and scoped native computer use for that dialog.

Preventing self-interruption

Early versions could hear Robin's own narration and treat it as participant speech. We added wake-word enforcement, echo comparison, playback state, and barge-in rules so Robin only stops or responds when a participant actually addresses it.

Keeping a live demo recoverable

Meet's interface and caption DOM changed during development. We added multi-selector parsing, persisted event traces, browser-recovery diagnostics, stale-tab protection, and regression tests based on the exact live failures.

Accomplishments we are proud of

  • Robin autonomously carried a live request from spoken delegation through grounded analysis, validation, a six-slide narrated presentation, clean shutdown, and a return to listening.
  • It recovered a spoken wake-word request through live Meet captions and replied through its BlackHole virtual microphone.
  • The GPT-5.6 agent produces multi-source, cited presentations rather than fixed finance templates.
  • The project offers a one-command operator workflow and a coherent real-time dashboard.
  • Safety boundaries, validation gates, approvals, emergency stop, and audit evidence are part of the product instead of afterthoughts.
  • The repository includes more than 225 core tests plus web tests, type checking, deterministic smokes, live API checks, and real-Meet rehearsal tooling.

What we learned

Building a proactive meeting agent is a systems problem. Strong reasoning is necessary, but initiative only feels real when browser state, operating-system permissions, audio devices, UI timing, source boundaries, recovery logic, and operator feedback all work together.

We also learned that proactivity feels much safer when it is socially and technically legible. Requiring Robin's name establishes authorization; independently doing the work demonstrates initiative; raising a hand preserves meeting etiquette; and citations plus visible state transitions keep Robin accountable.

Finally, live testing mattered more than isolated mocks. The most valuable improvements came from watching Robin fail in a real meeting, reading the resulting evidence, fixing the underlying state or integration problem, and adding a regression test.

What's next

  • Complete three consecutive fresh-start, second-participant rehearsals with every certification check passing.
  • Make native Chrome audio capture reliable across more macOS configurations so captions are a fallback rather than a requirement.
  • Expand grounded follow-up conversation and in-meeting revision workflows.
  • Add richer spreadsheet, document, and presentation editing while retaining workspace boundaries.
  • Improve named-speaker attribution when caption metadata is unavailable.
  • Extend Robin to calendars and additional meeting platforms.
  • Package the dedicated-Mac setup into a smoother installer for non-developer operators.

Built with

  • OpenAI GPT-5.6
  • OpenAI Responses API
  • OpenAI text-to-speech
  • OpenAI realtime transcription
  • Codex
  • Python
  • FastAPI
  • Pydantic
  • SQLite
  • Next.js
  • React
  • TypeScript
  • Playwright
  • Chrome DevTools Protocol
  • Swift
  • ScreenCaptureKit
  • AVFoundation
  • BlackHole 2ch
  • FFmpeg
  • pandas
  • python-pptx
  • Plotly
  • uv
  • pnpm

How judges can run it

Supported platform

The complete real-Meet workflow requires macOS, Google Chrome, BlackHole 2ch, Xcode Command Line Tools, Python 3.12+, Node 22+, uv, pnpm, and an OpenAI API key.

Setup

git clone https://github.com/piercebrookins/robin.git
cd robin
scripts/setup_partner.sh --real-meet --no-start
make robin

The setup script installs dependencies, creates .env, prompts for OPENAI_API_KEY, builds the Swift bridge, seeds demo files, and validates the environment.

Then:

  1. Open http://127.0.0.1:3000 if it does not open automatically.
  2. Paste a Google Meet URL.
  3. Select Join & listen.
  4. Keep Google Meet captions enabled.
  5. Say: “Robin, use the finance files to compare our 2024 quarterly results and make a few slides.”
  6. When Robin raises its hand, say: “Robin, you can share.”

The API documentation is available at http://127.0.0.1:8787/docs.

/feedback 019f724b-782d-7b30-8b3f-f97464bfd61a

Simulator and automated validation

Judges who do not want to configure a real Meet can still inspect and validate the core workflow using the documented simulator, fixture, smoke-test, and unit-test commands in the repository README.

Built With

Share this project:

Updates