Pocket Apps

Native apps for everyday ideas, running right on your phone.

Comment

Inspiration

I love building apps, and I love using them. AI has become an extraordinary accelerator for turning ideas into software—especially small, personal ideas that might never justify days of conventional development.

Products such as v0 and Lovable have made software creation remarkably accessible, but they are primarily hosted experiences. I wanted something more personal: an app builder I could carry, create with, and immediately use from my phone.

That question became Pocket: an app builder that lives on your iPhone.

For this hackathon, I wanted to push Pocket further. Building a useful native app is a substantial, asynchronous task involving planning, interface design, business logic, data modeling, testing, native previews, permissions, and installation. Instead of making the phone perform all of that agentic work locally, I built Pocket App Factory: a Google Cloud backend where specialized Gemini-powered agents can plan and build in parallel while the iPhone remains the trusted native compiler, preview environment, permission gate, and installer.

The idea is simple:

Cloud agents propose. The device certifies.

What it does

Pocket closes the distance between an everyday idea and an app you can actually touch and use.

A user describes an application in one conversational interface and can attach photographs, documents, or other reference material. Pocket sends the request, relevant project state, and the device’s current runtime contract to Pocket App Factory.

In Google Cloud, a coordinated group of bounded agents:

  • Plans the application as a versioned graph of views, actions, data, use cases, effects, and tests.
  • Builds declarative views and realistic preview states.
  • Implements business logic, persistence, schemas, migrations, and test scenarios.
  • Reconciles the resulting files into a source-only candidate.
  • Requests native previews and validation from the iPhone.
  • Reviews native screenshots with Gemini and proposes bounded repairs when necessary.

The phone remains authoritative. It downloads and verifies every source artifact, compiles the TypeScript, renders the actual SwiftUI interface, executes production-like scenarios, asks the user to approve sensitive capabilities, seals the checkpoint, installs the app, and launches it.

Pocket Apps can use native capabilities such as the camera, photo library, location, files, sharing, notifications, and on-device databases. Apps that do not require a network or an agent can continue running locally after installation.

The experience still feels like one conversation. The user does not have to manage a visible swarm of agents, even though planning, interface construction, logic, and testing can proceed independently in the cloud.

How we built it

Pocket’s format-v3 app runtime, dynamic SwiftUI renderer, embedded TypeScript compiler, local LLM builder, capability system, and generated-app format existed before this hackathon.

The new hackathon work is Pocket App Factory: the Google Cloud backend, production remote transport, multi-agent workflow, durable recovery system, source-validation layer, cost controls, and their integration with the existing iOS experience.

The backend uses:

  • Gemini Flash through Vertex AI for planning, structured generation, repair, and multimodal screenshot review.
  • Google ADK to define bounded Planner, View Builder, Logic/Test Builder, and Finalizer roles.
  • Cloud Run for the public gateway, private workers, and deterministic validation service.
  • Firestore as the authoritative journal for sessions, operations, event sequences, graph revisions, work leases, idempotency records, candidates, and model-cost accounting.
  • Pub/Sub for duplicate-tolerant, event-driven work dispatch.
  • Cloud Storage for attachments, generated source artifacts, native screenshots, and diagnostic bundles.
  • Cloud Logging for correlated evidence across every operation, work unit, candidate, graph revision, and model invocation.

The gateway is stateless. WebSockets stream events from Firestore after the client’s last acknowledged sequence, so reconnecting does not depend on returning to the same Cloud Run instance. If Pocket is backgrounded or terminated, the cloud build continues. When the app returns, the client reconnects, replays missing events, restores the conversation, and resumes from the latest durable candidate or checkpoint.

Workers acquire transactional leases, produce scoped file proposals, and commit their results idempotently. Shared and contested files are reconciled by the Finalizer rather than being edited concurrently. Deterministic validation—not an agent—enforces file ownership, paths, schemas, capabilities, digests, runtime compatibility, and publication barriers.

I continued the recursive development constraint from the original Pocket project: a local agent performed the implementation while I directed the architecture, product decisions, experiments, debugging, and evaluation.

Challenges we ran into

The largest challenge was deciding where trust should live.

The cloud is good at planning, parallel reasoning, generation, and repair. The phone is the only environment that can faithfully answer whether the generated app compiles against Pocket’s current declarations, renders correctly in the native runtime, respects capability permissions, survives real lifecycle transitions, and can safely be installed.

That led to a deliberately asymmetric architecture: the cloud can propose source code and request host actions, but it cannot generate trusted compiled artifacts, approve permissions, seal checkpoints, or publish an application.

Durable recovery was another major challenge. Cloud Run WebSockets are temporary, Pub/Sub delivery can be duplicated, iOS can suspend or terminate the client at any time, and a user may leave the builder to use another Pocket App while generation continues. Every command therefore has a stable identity, every server event has a monotonically increasing sequence, and both sides preserve enough state to resume without duplicating a user request.

Generated-code reliability also required more than better prompting. Early runs exposed issues that looked like model failures but were really incomplete contracts or validation gaps. For example, an interactive camera flow was valid TypeScript but inherited a 15-second use-case deadline, causing the native runner to terminate it while the user was still operating the camera. The solution was not another prompt attempt: Pocket now derives async deadlines from the client-supplied runtime profile and rejects invalid deadlines before publication.

Cost became an equally important engineering constraint. Agentic fan-out, large runtime declarations, retries, and multimodal review can become expensive quickly. The backend now records model usage per operation, limits provider calls and repair revisions, uses deterministic assembly whenever the application graph is complete, and enforces a project-wide monetary kill switch before starting another paid model call.

Accomplishments that we’re proud of

Pocket now has a real remote app-generation architecture rather than a canned backend demonstration.

A user can submit an open-ended request through the existing Pocket conversation, receive durable progress while cloud work continues, download verified source candidates, render native previews on the iPhone, return screenshot evidence to Gemini, run local validation, approve capabilities, install the checkpoint, and launch the resulting application.

The architecture preserves a single coherent user experience while allowing planning, view construction, logic, and testing to proceed in parallel where their contracts permit.

The Book Atlas evaluation generated a personal book-library application with an on-device database, categories, search, ratings, camera and photo-library integration, and an agent-backed metadata workflow. In Simulator, the installed app successfully created and persisted a real book, displayed its details, survived leaving and reopening the app, and returned the book through search. Exercising the actual camera path also uncovered the async-timeout defect that led to a platform-level validation improvement.

I am particularly proud that failures produced reusable improvements to the platform rather than one-off patches to the generated app.

What we learned

The first lesson was that parallel agents need explicit contracts more than they need unlimited context.

The Planner must define stable view inputs, output actions, mock states, use-case contracts, effects, data ownership, dependencies, tests, and file ownership before fan-out becomes reliable. Workers perform better when they receive a small, authoritative assignment instead of the complete user conversation.

The second lesson was that deterministic infrastructure is what makes agentic systems trustworthy. Agents can plan, create, critique, and repair, but leases, retries, event ordering, file ownership, digest verification, capability enforcement, and publication must remain host-controlled.

The third lesson was that native feedback is enormously valuable. A Python validator can catch structural and TypeScript errors cheaply, but only the phone can certify the complete application. Returning real native screenshots and diagnostics to Gemini creates a much stronger repair loop than asking the model to review its own source in isolation.

Finally, faster builds do not come only from using a faster model. The largest improvements came from reducing unnecessary model calls, validating locally before requesting a repair, assembling deterministic artifacts without a Finalizer call when possible, reusing cached runtime context, and retrying only the affected shard.

What’s next for Pocket

The immediate goal is to make the complete remote creation path reliable enough for broader testing: more holdout applications, repeated end-to-end runs, stronger lifecycle recovery, and measured comparisons between parallel and forced-serial scheduling.

The next optimization is economic. Pocket App Factory should route simple and deterministic work away from expensive model calls, use smaller models for bounded roles, cache stable runtime context, and reserve high-capability reasoning for planning and difficult reconciliation.

Beyond the hackathon, Pocket could become the place where a software idea begins, is tested, and evolves. A Pocket App might remain a small personal tool, be shared with another Pocket user, or eventually graduate into an independently packaged iOS application.

The long-term vision remains the same: make software creation available at the moment an idea appears, on the device where that software will actually be used.

Built With

Share this project:

Updates