Coco

About the project

COCO, short for Cognitive Concierge Operator, is a voice-first Android assistant that can operate existing mobile applications on a user’s behalf.

A user can say “Hi Coco” and describe an outcome such as:

“Find me a highly rated paneer pizza under ₹500 and add it to my cart.”

COCO listens to the request, understands the intent, opens the appropriate application, observes the current screen, decides the next action, and interacts with the interface through Android Accessibility. It continues this observe–reason–act cycle until the task is complete or a decision requires the user’s input.

Unlike traditional mobile automation, COCO does not depend on a fixed sequence of coordinates or assume that every session begins on the same screen. Each action is based on the interface currently visible to the user. If the screen changes, an expected result does not appear, or the task becomes ambiguous, COCO can reassess the situation instead of blindly replaying a script.

COCO is designed to make everyday mobile tasks more accessible and less repetitive while keeping consequential decisions visible to the user. It can help navigate search and shopping flows, compare available options, handle required selections, and prepare a cart. It asks the user when preferences are unclear and requires explicit authorization around sensitive actions such as payment setup or completing a purchase.

Inspiration

The inspiration for COCO came from a simple frustration: voice assistants can often answer questions or open an application, but the user still has to complete most of the actual work.

A request such as “order dinner” expands into many small interactions—choosing an app, searching, filtering results, comparing restaurants, selecting an item, handling customizations, reviewing the cart, and confirming payment. These steps are individually simple, but together they demand time, attention, vision, and familiarity with constantly changing interfaces.

Fixed automation was not a satisfying solution. A recorded macro breaks when a button moves, a dialog appears, an application changes its layout, or the user starts from an unexpected screen. We wanted to explore a different question:

What if a mobile assistant could operate an interface the way a careful person does—by continuously observing what is actually on the screen?

That question became COCO. Its central principle is:

$$ \text{Observe} \rightarrow \text{Reason} \rightarrow \text{Act} \rightarrow \text{Verify} \rightarrow \text{Observe again} $$

The goal is not to remove the user from the process. The goal is to remove repetitive interaction while preserving the user’s judgment, privacy, and authority.

How we built it

COCO is built as three cooperating layers.

1. Flutter application

The Flutter layer provides the main application interface, command entry, settings, execution status, and controls for stopping a task. It also connects the cross-platform interface to Android-native services through platform channels.

2. Android operator runtime

The Android/Kotlin layer performs the device-side work:

• listening for the “Hi Coco” wake phrase;
• capturing speech through a shared microphone pipeline;
• providing on-device speech recognition as a fallback;
• observing interface elements through Android Accessibility;
• launching applications;
• executing taps, text input, scrolling, and navigation actions;
• displaying a floating bubble and decision interface over the active app; and
• masking sensitive screen regions before optional visual reasoning.

The floating interface helps users understand what COCO heard, what it is currently doing, and when it needs assistance.

3. Python reasoning backend

A FastAPI service receives the user’s goal and a structured representation of the current screen. It maintains the task state and selects one next action.

The backend includes:

• intent and application selection;
• semantic screen understanding;
• action planning and confidence routing;
• candidate gathering and evidence-based ranking;
• task, cart, and pending-decision state;
• action authorization and payment-safety checks;
• verification of whether an action produced its expected effect;
• recovery when the interface is stuck or unexpected; and
• encrypted memory for useful screen, element, and task knowledge.

The operator does not treat a generated plan as permanent truth. Fresh observation remains authoritative. After an action, COCO checks whether the expected interface change occurred before it commits progress or announces success.

Challenges we faced

Maintaining one coherent decision

One of the hardest problems was keeping a decision consistent across the backend and Android client. A candidate could be ranked correctly by the server but then be recomputed, consumed more than once, or interpreted differently during actuation.

We addressed this by introducing a canonical decision object with an explicit lifecycle. Evidence produces a decision once, the client consumes it once, and subsequent progress depends on a newly observed screen rather than repeatedly using stale intent.

Knowing whether an action actually worked

A successful tap gesture does not guarantee a successful outcome. The target may have moved, the app may have ignored the gesture, or a loading state may have prevented the interaction.

COCO therefore predicts the expected effect of an action and checks the next observation for evidence of that effect. When the effect is absent, it can retry safely, choose an alternative target, replan, or ask the user. This prevents the system from reporting false success simply because it issued a command.

Understanding natural voice replies

Voice interaction is rarely a sequence of complete, perfectly phrased commands. Users reply with short answers such as “the second one,” “yes, that’s fine,” or “without onions.” Silence can mean cancellation, hesitation, or that the user did not hear the question.

We learned that replies must be interpreted in the context of the pending decision. COCO maintains that conversational state so a reply can resolve the current ambiguity instead of being mistaken for an unrelated new task.

Working with dynamic third-party interfaces

Accessibility trees can be incomplete, noisy, or structured differently across applications. Visible labels are not always attached to the element that actually receives a tap. Some controls appear only after scrolling, while others are visually present but absent from the semantic hierarchy.

To handle this, COCO combines structural accessibility information with element labeling, target scoring, screen identity, action history, and optional privacy-masked visual escalation. It also uses repeat guards and recovery logic to avoid endlessly performing the same ineffective action.

Balancing autonomy with safety

An operator that can interact with shopping, messaging, settings, and payment interfaces needs clear boundaries. Too many confirmations make the assistant frustrating; too few make it unsafe.

We separated ordinary reversible navigation from consequential actions. COCO can search, compare, and prepare, but ambiguous selections are surfaced to the user and sensitive commitments require explicit authorization. Payment information is never treated as an ordinary text field.

What we learned

The biggest lesson was that a useful agent needs more than strong language understanding. Reliability comes from disciplined state management.

The system must always be able to answer:

• What is the user trying to achieve?
• What does the current screen prove?
• Which decision is active?
• Has that decision already been used?
• What effect should the last action have produced?
• Did that effect actually occur?
• Is it safe to continue without asking the user?

We also learned that uncertainty should be treated as a product feature rather than hidden as a model failure. A concise question at the correct moment is better than a confident but incorrect action.

Voice introduced another important lesson: speech recognition confidence is not the same as task confidence. COCO may hear every word correctly and still need clarification about the user’s preference. Conversely, when transcription confidence is weak, confirming the command before acting prevents an error from propagating through the entire task.

Finally, building across Flutter, Kotlin, Android Accessibility, audio processing, and a Python reasoning service reinforced the importance of end-to-end traces. Debugging only one layer is insufficient. We needed to follow a decision from the original voice request, through server reasoning, into Android actuation, and back into post-action verification.

What we are proud of

COCO now demonstrates a complete voice-to-action loop on a physical Android device. It can wake from a spoken phrase, understand a goal, operate an existing application dynamically, expose its current state through a floating interface, recover from uncertainty, and stop at meaningful human-authorization boundaries.

We are especially proud that the project is not merely a chat interface attached to a collection of fixed automations. Its architecture treats the live interface as the source of truth and is designed around verification, recovery, privacy, and user control.

What’s next

The next stage for COCO is to make the operator easier to test and safer to use across a wider variety of devices and applications. Planned work includes:

• expanding automated replay evaluations across more applications and interface states;
• measuring task completion, recovery, and user-intervention rates;
• improving multilingual and accented speech support;
• providing a user-facing privacy and memory dashboard;
• packaging a simpler installation and hosted test experience; and
• extending the same observation-first architecture to more everyday workflows.

My long-term vision is for COCO to become a personal action layer for the phone: integration-optional, accessibility-first, transparent about uncertainty, and always bounded by the user’s authority.

Built With

• flutter
• kotlin
• python