PhoneClaw: Local AI Agent for Your Phone

Inspiration

Phones are becoming powerful enough to run useful AI locally, but most AI agent products still treat the phone as a thin client: a screen, a notification endpoint, or a remote controller for cloud agents.

I built PhoneClaw because I want the phone itself to become the AI agent runtime: private, mobile-native, permission-aware, and able to use real device capabilities like Calendar, Reminders, Contacts, HealthKit, camera, voice, widgets, Live Activities, and on-device models.

PhoneClaw is already a TestFlight-ready iPhone app and open-source project with 1k+ GitHub stars. For OpenAI Build Week, I used Codex and GPT-5.6 to meaningfully extend and harden the product’s local agent path: better Gemma 4 tool-calling, native Skill execution, prompt fidelity, validation, and the path from natural language to real phone actions.

What it does

PhoneClaw turns an iPhone into a local AI agent.

It runs local models on-device, routes natural-language requests into native iOS Skills, and supports real phone tasks across Calendar, Reminders, Contacts, Clipboard, HealthKit, image understanding, voice, web search, Live mode, LiveLand, and optional Mac Gateway inference.

Users can ask PhoneClaw to understand context, call the right native Skill, clarify ambiguous requests, and execute actions while keeping personal data local by default.

For OpenAI Build Week, I focused on improving the product-level agent loop:

• Stronger Gemma 4 prompt and tool-call fidelity on the LiteRT path
• More reliable native Skill execution from natural language
• Better handling of clarification, tool results, and multi-turn context
• Validation fixtures and tests for real mobile-agent behavior
• Clearer documentation of what existed before Build Week and what was extended during the hackathon

How I built it

PhoneClaw is built with Swift, SwiftUI, native iOS frameworks, local model runtimes, LiteRT/Gemma 4, MiniCPM-V, native Skills, Live/LiveLand interaction surfaces, and an optional Mac Gateway for local-network inference.

I used Codex and GPT-5.6 as the main development partner during Build Week. Codex helped inspect the existing architecture, identify the highest-impact product work, implement the Gemma 4 / LiteRT agent improvements, add validation coverage, debug edge cases, and keep the project aligned with the rule that existing projects must be meaningfully extended during the submission period.

The result is not a prototype from scratch. It is a product-level iPhone AI agent app, extended during Build Week with Codex/GPT-5.6 to make the local mobile-agent loop more reliable, judgeable, and closer to production release.

Challenges

The hardest part was drawing the right boundary. PhoneClaw already existed as a serious app, so the submission could not simply be “here is my existing project.” I needed to identify the Build Week work that meaningfully improved the product and could be judged on its own.

Another challenge was reliability. Mobile-native agents are harder than chat demos because they need to convert natural language into real device actions, respect permissions, handle ambiguity, preserve context, and avoid unsafe or accidental tool calls.

I also had to make the project understandable for judges. A local iPhone agent involves device setup, model runtimes, native Skills, TestFlight builds, and real mobile constraints, so I focused on documentation, evidence, and a clear demo path.

What I learned

Codex is most valuable when working inside an existing, complex codebase. It helped me move from product goals to concrete implementation while preserving the architecture that already worked.

I also learned that mobile AI agents need more than a model. They need a runtime, native capabilities, permission boundaries, clarification logic, validation, and product surfaces that make the agent useful in daily phone workflows.

Build Week helped turn PhoneClaw from “a local AI agent app that works” into a more disciplined product submission: clearer scope, stronger validation, better Gemma 4 tool-calling, and a more reliable path from user intent to real phone action.

Built With

• activitykit
• agents
• ai
• bonjour
• cli
• codex
• contacts
• developer
• eventkit
• gemma
• gpt-5.6
• healthkit
• ios
• litert
• minicpm-v
• mobile
• ollama
• on-device
• openai
• swift
• swiftui
• testflight
• tools
• widgetkit
• xcode