-
-
Android Relay
-
Agent Dashboard
-
Agent interacting with alarms
About the project
Android Relay gives AI agents a bounded, evidence-first way to interact with real Android devices.
The project grew out of Android development work I was doing at Square. As AI became more capable at writing and reasoning about code, one bottleneck became especially obvious: testing Android changes. Agents could help build features, but they struggled to efficiently interact with physical devices and emulators to validate the result. Existing approaches were often slow, brittle, app-specific, or unable to prove that an action actually produced the intended user-visible outcome. I wanted to build a tool that allows for us to interact with Android devices efficiently, to help to allow AI to test and identify issues in our software.
Android Relay is a local Android controller that connects to one explicitly selected, ADB-authorized device. It captures screenshots and UI hierarchy data, then exposes generic MCP tools for observing the device, finding UI elements, clicking, typing, launching apps, waiting for state changes, and verifying outcomes. It is not built around hardcoded flows for Discord, Clock, or another individual app; it works from accessibility metadata such as visible text, descriptions, resource IDs, hierarchy context, and editable fields.
The core principle is that command dispatch is not proof. Before an action, Android Relay refreshes and validates the UI target, rejecting ambiguous selectors. Before text entry, it proves the exact destination field. After an action, it looks for independent evidence in the resulting UI whenever possible. If the screen is ambiguous or the intended result cannot be verified, it stops and records the uncertainty instead of continuing to guess.
Latency was a major technical challenge. Full screenshot-and-hierarchy observations and repeated agent-to-tool round trips made simple device interactions too slow. We added hierarchy-only observations, compact tool responses, cached launcher resolution, atomic text-entry verification, and server-side multi-step navigation. This allows the controller to resolve a known route, prove the destination field, submit text, and verify the result in one bounded operation.
Android Relay also creates an auditable evidence trail for every run: screenshots, UI observations, actions, verification results, timing data, and an exportable report. A local command-center dashboard streams the tablet view and action timeline, with pause and stop controls for human intervention.
We validated the current local alpha on a Samsung Galaxy Tab S8 using existing Android apps and system UI. Android Relay is intentionally scoped rather than claiming universal Android automation: the goal is a controlled, efficient real-device testing loop that lets AI agents help close the gap between writing Android code and confidently validating it.
Log in or sign up for Devpost to join the conversation.