Inspiration

AI agents have evolved significantly. Today, they can browse the web, write code, operate computers, and complete complex tasks autonomously using computer-use capabilities. However, we noticed a major gap: while AI agents can interact with desktop computers, they still have limited ability to directly operate smartphones in the same way.

A huge part of our digital lives exists on our phones. From messaging and managing notifications to using productivity tools and other mobile applications, many everyday tasks happen primarily on Android. Yet, most AI assistants cannot directly navigate these applications, interact with their interfaces, or complete workflows on our behalf.

Existing alternatives often rely on cloud-hosted browsers, external integrations, or third-party services. These approaches may require users to log in again, share authentication credentials, or connect sensitive accounts to external platforms.

What if an AI agent could simply use the applications already installed and logged in on your phone, just as a human would?

That question inspired us to build Pocket Agent.

Our goal was to bring computer-use capabilities directly to Android, allowing AI agents to see the screen, tap, swipe, navigate applications, and perform real tasks without requiring users to share their app passwords or authenticate through unfamiliar cloud-hosted sessions.

But we wanted to go beyond basic phone automation.

We envisioned a Jarvis-like personal assistant that acts as the central point of communication while delegating tasks to multiple specialized AI agents. These agents could operate independently in background virtual displays, allowing users to continue using their phones normally while the agents work.

Our vision is to make smartphones as accessible to AI agents as computers are today—turning AI from something that merely provides instructions into something that can actually execute them on your own device.

What it does

Pocket Agent is an autonomous, multi-agent AI assistant for Android that can interact with real applications and perform tasks directly on your smartphone.

Unlike traditional AI assistants that primarily answer questions or provide instructions, Pocket Agent can take actions through Android’s native capabilities and graphical interfaces.

Its key features include:

Real Android Computer Use: AI agents can observe the screen, tap, swipe, scroll, open applications, and navigate user interfaces to perform tasks.
Multi-Agent Orchestration: A central AI assistant can spawn specialized subagents, delegate tasks, monitor progress, and receive results while continuing its conversation with the user.
Parallel Background Execution: Up to three agents can independently interact with applications using separate virtual displays, without taking over the phone’s main display.
Screen-Off Operation: On supported Android configurations, background agents can continue operating virtual displays even when the physical screen is turned off.
Natural Voice Interaction: Users can communicate through voice using Android speech recognition and Deepgram for speech processing and responses.
Existing App Sessions: Agents interact with applications already authenticated on the device, avoiding the need to provide individual app credentials to a remote browser automation service.
Persistent Tasks and Scheduling: Agents can manage longer-running tasks, preserve execution state, and schedule work for later.
Extensible AI Capabilities: Support for multiple AI providers, per-agent model selection, and external Model Context Protocol (MCP) tools.

For example, instead of manually opening multiple applications, gathering information, and switching between them, a user could ask Pocket Agent to perform the workflow.

The main assistant would understand the request, delegate the necessary operations to subagents, and communicate the results.

The key difference is that Pocket Agent is not just a chatbot with phone controls. It is an orchestration system where one assistant coordinates multiple autonomous agents operating on a real Android device.

How we built it

We built Pocket Agent as a native Android application combined with an embedded AI agent runtime, allowing the orchestration system and phone-control tools to operate directly on the device.

The architecture consists of several components:

Multi-Agent Runtime

We integrated the Pi agent framework and an adapted @pi9/subagent runtime through Javet, which embeds a Node.js execution environment inside the Android application.

A primary agent manages the user conversation and delegates tasks to independent workers. Each worker maintains its own context, preventing complex background workflows from unnecessarily consuming the main conversation’s context.

Android Computer-Use System

We implemented native Android tools using Accessibility Services and Android system capabilities.

For background interaction, we integrated Shizuku-backed virtual displays, allowing multiple agents to operate in separate display environments on supported devices.

A shared display pool manages up to three concurrent virtual displays and coordinates access between workers.

Voice Interaction

We integrated Deepgram for speech recognition and synthesis alongside Android’s native speech-recognition capabilities.

The application supports multiple speech-recognition modes, including a hybrid approach that combines Android and Deepgram recognition to balance responsiveness and accuracy.

Persistent Agent Orchestration

We implemented persistent agent conversations, execution checkpoints, and a task-event system.

Instead of requiring the main assistant to constantly poll workers, completed tasks can report their results through a persistent event inbox.

SQLite stores conversation and execution state, helping the application manage interruptions and longer-running workflows.

AI Providers and Extensibility

Pocket Agent supports multiple AI inference providers, including Codex account integration and compatible API-based providers.

We also integrated MCP, allowing the assistant to access additional external tools alongside native Android capabilities.

Sensitive provider credentials are stored using Android Keystore-backed encryption.

Throughout development, we focused on keeping phone interaction and agent orchestration inside the Android application rather than depending on a remote desktop or cloud-hosted phone.

Challenges we ran into

Running AI agents in the background

One of our biggest challenges was Android’s restriction on background execution and graphical interaction.

Unlike desktop operating systems, Android is not designed for multiple independent automation agents to control different application interfaces simultaneously.

Implementing virtual-display-based computer use required dealing with Android permissions, application launching, accessibility interactions, and differences between device configurations.

Making multiple agents work simultaneously

Running one computer-use agent is challenging. Running several agents independently introduces another level of complexity.

We had to manage:

Allocation and reuse of virtual displays.
Isolation of worker conversations and execution state.
Communication between the main assistant and subagents.
Cancellation, interruption, and completion of background tasks.
Preventing agents from interfering with one another.

Designing the orchestration system to remain responsive while background tasks execute was a significant engineering challenge.

Building reliable voice interaction

We wanted Pocket Agent to feel like a natural personal assistant rather than a traditional chatbot.

However, voice interactions introduced challenges involving recognition latency, microphone availability, speech playback, interruptions, and transitions between listening and speaking.

Coordinating these components while agents execute tasks in the background required careful lifecycle management.

Handling long-running tasks and failures

Real-world phone automation is unpredictable.

Applications can load slowly, interfaces can change, permissions may be missing, and Android can interrupt background processes.

We needed a system capable of preserving task state, reporting failures, and avoiding unintended repetition of actions after interruptions.

Android device compatibility

Android behavior varies across manufacturers, operating-system versions, and permission configurations.

Some background-display features depend on Shizuku and device-specific behavior. Ensuring consistent execution across physical phones remains an ongoing challenge.

Accomplishments that we’re proud of

Our biggest accomplishment is bringing together real Android computer use and multi-agent orchestration within a single application.

Some milestones we are particularly proud of include:

Building an embedded multi-agent runtime: The main assistant and its subagents run inside the Android application without requiring a separate orchestration server.
Implementing parallel phone interaction: Multiple agents can use independent virtual displays, enabling background tasks without monopolizing the primary screen.
Creating a Jarvis-like architecture: Users communicate with one assistant while specialized background agents handle delegated work.
Supporting real application interaction: Agents can operate Android interfaces using taps, swipes, screen observation, and native device tools.
Integrating natural voice conversations: Combining speech recognition, synthesis, and agent execution makes the assistant more accessible.
Preserving user control: Users can inspect ongoing tasks, interrupt workers, and manage background execution.
Avoiding unnecessary credential sharing: Agents can use existing authenticated Android app sessions without requiring users to provide those app passwords to a separate cloud automation platform.

We are especially proud that this architecture moves beyond conventional chatbot-based automation.

Rather than relying entirely on APIs or predefined integrations, Pocket Agent gives AI agents a way to interact with the applications people already use.

What we learned

Building Pocket Agent taught us that creating a truly autonomous AI assistant requires much more than connecting a language model to tools.

We gained experience in several areas:

Multi-Agent Systems: Designing independent workers, coordinating execution, managing context, and communicating results between agents.
Android Internals: Understanding accessibility services, virtual displays, background execution, permissions, and application lifecycle behavior.
Concurrency and Resource Management: Safely allocating shared resources across multiple agents executing simultaneously.
Reliable Agent Execution: Preserving task state, handling interruptions, and managing asynchronous workflows.
Voice AI: Integrating speech recognition and synthesis while balancing responsiveness and reliability.
Security and Privacy: Understanding the importance of keeping application authentication on-device and minimizing unnecessary exposure of sensitive credentials.
Real-World AI Reliability: Learning that a model’s ability to reason is only one part of successful automation. The surrounding system must handle unpredictable interfaces, failures, and incomplete actions.

One of our most important realizations was that the real challenge is not making an AI agent perform a single action, but enabling multiple agents to execute complex workflows reliably while the user remains in control.

What’s next for Pocket agent

Our long-term goal is to make Pocket Agent a genuinely useful personal assistant capable of managing everyday smartphone workflows with minimal supervision.

We plan to focus on several improvements:

More reliable Android computer use

Improve compatibility across Android devices, handle unexpected UI changes more gracefully, and strengthen background execution.

Smarter multi-agent coordination

Enhance task delegation, agent collaboration, scheduling, and recovery so that more complex workflows can be completed autonomously.

More natural voice interaction

Continue reducing voice latency, improving interruption handling, and making conversations feel closer to interacting with a real personal assistant.

Better memory and personalization

Improve the assistant’s ability to remember relevant preferences, understand recurring workflows, and adapt to how individual users interact with their phones.

Greater privacy and user control

Strengthen permission management, approval workflows, and transparency around agent actions while minimizing unnecessary sharing of sensitive information.

More powerful real-world workflows

Expand support for complex tasks involving multiple Android applications, background agents, and external tools.

Ultimately, we want to close the gap between what AI agents can do on computers and what they can do on smartphones.

Our vision is simple: You should be able to tell your phone what you want to accomplish, and Pocket Agent should be able to figure out how to get it done.

Instead of an AI that only tells you what to do, we want to build an AI that can actually do it for you.

GitHub: https://github.com/ru-aish/pocket-agent