Granular

Inspiration

We started Granular from an observation that 90% of our questions with ChatGPT (and other LLM providers) has incredibly long resolution time because the answer arrives as a static wall of instructions while the real work happens in another tab. You become the messenger between the instructions and the interface as every tab-switch means re-describing your status, pasting screenshots, and hunting for your place in step 4 of 9.

When the team saw GPT-5.6-sol release demo with sol correct a users grammar in real time without waiting until the conversation was finished, the gap became obvious: proactive intelligence that follows along and speaks up at the right moment is what's missing from today's human-LLM interfaces. And so granular, the resolution we think agents should operate at, became both the name and the design principle for this project.

What it does

Granular is a new interface between humans and LLM agents that enables continuous interaction with the intelligence layer itself. It replacing existing, interruptive channels such as turn-based chat windows and hotkey-activated agents with a persistent agent observation loop that grounds every exchange in the user's live screen and intervenes only when needed.

Granular's agent Opal accepts any text-based instructional query, such as "How do I export my current Codex session ID?", and compiles it into a sequence of modular steps with explicit completion criteria and per-step progress/state attribution called the task graph. Launching the graph in Cruise mode starts a hands-free (no 'fn' key needed!) audio stream between the user and the agent that runs for the entire task. Ask about any element on screen ("which of these is the session ID?") and Opal responds in voice while directing a click-through ghost cursor at the exact target, grounded in the active step and the live interface.

When progress stalls, deviates, or breaks, Opal intervenes immediately without prompting. Once task-completion is observed, Opal ends Cruise mode itself: audio and observation shut down automatically, raw audio and screenshots are neither uploaded nor retained, and only sanitized local execution diagnostics persist for debugging.

How we built it

Granular is build on a TypeScript monorepo that holds the shared Zod contracts, a deterministic Cruise engine (pure reducer, replayable JSONL fixtures), and a Cloudflare Worker that fronts GPT-5.6-sol through structured outputs for chat, two-stage graph compilation, stall evaluation, and grounded Q&A.

The native SwiftUI app carries a line-for-line Swift port of that engine (fixture parity with the TS version enforced in CI) plus ScreenCaptureKit and Accessibility observation, passive activity monitoring that never reads keystroke contents, ElevenLabs realtime STT over WebSocket with single-use tokens, streaming TTS, and click-through overlay panels for the cursor halo and ghost pointer. Opal, is powered by gpt-5.6-sol.

Challenges we ran into

Rigorously defining Opal's workflow and harness so that progress detection never degenerates into token-bashing was definitely the hardest part of building granular. We ended up inverting the architecture: a deterministic local engine owns progress attribution by evaluating each step's completion predicates against Accessibility state and semantic page hashes, screen capture is demoted to a single local 1fps keyframe that never leaves the machine, and voice commits only finalized transcripts, so Opal is only queried under three conditions: graph compilation, a confirmed stall, or a user question.

Accomplishments that we're proud of

The idea itself is the first thing that we're really excited about of because of how much it brings users closer to the models that already accelerates their day-to-day workflow.

What we learned

Proactive agents live or die on evidence: an explicit task graph with completion predicates beats any amount of prompting for knowing when to stay silent, which is 90% of the product. Also, LLMs are a lot more reliable when they output semantics and deterministic code owns the schema (too many tokens were burned trying to one shot the task-graph compilation process...).

What's next for Granular

Product direction: today Opal only accepts text-native queries, but instructions don't live only in text. The end goal is compiling any representation of a workflow (a tutorial video, a company's internal onboarding markdown, a compliance checklist) into the same task graph, so anything that encodes "how to do something" becomes a guided, observed, hands-free session. Your company's tribal knowledge shouldn't need rewriting to become interactive.

Features wise: cross-platform OS-native clients beyond macOS, an auto-paste feature for hand-off of values like names and IDs, and full cloud backend integration with real accounts to process actual user transactions.

Built With

• cloudfare
• codex
• elevenlabs
• openai
• sqlite
• swift
• typescript