Aura

I gave your phone eyes 👀 and hands 🖐️ Aura turns Android into an agentic substrate for multimodal perception and interaction. Its on-device MCP layer exposes device capabilities to external agents

Comment

Inspiration

I gave your phone eyes and hands.

For over a decade, smartphones have become extraordinarily capable, yet the fundamental interaction model has remained remarkably unchanged: tap, swipe, search, repeat.

The recent emergence of multimodal models and agentic systems raised a more interesting possibility: what if an AI could operate the device itself, rather than merely instructing a human on how to use it?

That question became Aura.

What is Aura?

Aura is an AI-native control layer for Android, designed to give AI agents a grounded interface to the device.

It combines multimodal perception, reasoning, memory, and interaction into an agentic loop that can observe the device, determine the appropriate course of action, execute it, and verify the outcome.

More importantly, Aura is not confined to a single agent.

It incorporates an on-device MCP server, allowing external AI agents to discover and invoke capabilities on the phone through a standardised interface.

In effect:

AI Agent → MCP → Aura → Android

How I Built It

Aura is built natively for Android using Kotlin, multimodal AI, Android Accessibility, and the Model Context Protocol (MCP).

At its core is an iterative observe → reason → act → verify loop.

Rather than assuming that an action succeeded, Aura continually reassesses the device state and adapts to the result. This makes the system considerably more resilient to dynamic interfaces, unexpected states, and variations between applications.

A dedicated memory layer provides continuity across interactions, allowing relevant context to persist beyond a single task.

The MCP layer extends this architecture beyond Aura itself, exposing the device as an environment that other agents can interact with.

What I Learned

Building Aura shifted my perspective on what an AI agent actually requires.

Generating an action is relatively straightforward. Reliably executing that action in an uncontrolled, changing environment is the difficult part.

I learned that perception, verification, recovery, and memory are not auxiliary features of an agent; they are fundamental to making one useful.

I also found that mobile environments introduce an interesting constraint: the agent must operate within the latency, compute, and interface limitations of an actual consumer device rather than an abstract software environment.

Challenges

The principal challenge was reliability.

Android applications are inherently dynamic. Interfaces change, elements move, loading states intervene, permissions appear, and identical tasks can present different UI states.

A rigid sequence of predetermined actions therefore breaks easily.

Aura instead treats interaction as a feedback loop: observe the current state, reason about it, act, and inspect the result before proceeding.

Another challenge was designing memory without simply accumulating context. An effective agent must distinguish between information that is genuinely useful and information that merely adds noise.

Finally, exposing device capabilities to external agents through MCP required a careful boundary between agent autonomy and device control, while keeping the interface sufficiently general to support different agents and workflows.

What's Next

Aura is an exploration of what a more agentic computing interface could look like.

Rather than building another assistant that talks about what your phone can do, the aim is to create a layer through which AI can actually operate it.

The longer-term vision is simple:

Your phone shouldn't merely be smart. It should be usable by the intelligence you choose to put on it.

Built With

Share this project:

Updates

Submission history