Inspiration

In-person experiences have become harder to experience with intentionality. Even when meeting with friends, we're often scrolling on short-form content rather than truly engaging with those around us in the physical world. But what if technology could bring us together in the physical world instead of keeping us glued to our screens? We built Pear to explore ways to bring entertainment off the screen and back into the real world.

What it does

Library Quick Draw

Pear is a platform for creating and playing movement-based games using Apple peripherals such as iPhones, Apple Watches, and AirPods.

Pear repurposes devices people already own as spatial inputs for interactive experiences instead of using dedicated motion-tracking hardware.

Pear consists of two interconnected applications:

  • Pear Studio: A desktop development software that allows creators to develop and test games without needing to own every supported Apple device.
    • 3D device simulator where developers can add virtual peripherals, configure their inputs, and test how games respond to player movement.
    • Client-side scripts written in Javascript handling player inputs and server-side scripts managing shared multiplayer state.
    • A Plugin system allows developers to expand Studio's capabilities, including our integration of LangChain Deep Agents, which provides agentic AI-assisted development.
  • Pear Player: Players can connect their peripherals and interact with games through their movements and physical environment.

How we built it

Studio Library Studio Project Editing

Pear Studio is our desktop development environment. It has a 3D workspace where developers add virtual peripherals, write game logic in JavaScript, and test without owning every device. Client scripts handle per-player input, server scripts own shared multiplayer state, and remote events connect the two. A local test server lets developers run a game with simulated players before publishing.

One input format for every device. Real devices (read through Apple's sensor frameworks) and the simulated ones in Studio's 3D simulator emit the same standardized JSON input, so a game doesn't care what hardware is underneath.

Plugins, and LangChain Deep Agents. Studio has a plugin system, and our flagship plugin is a Python runtime built on LangChain Deep Agents. We use it in three ways:

  • Authoring agent. From the Deep Agent panel, a deep agent edits a project's scripts, scene and game instructions. It uses Deep Agents' planning (a todo list), two sub-agents (one for client JS, one for server JS), and a project-scoped filesystem backend. Writes are limited to the permissions the plugin manifest declares and can't escape the project folder. It loads a Pear scripting skill and an AGENTS.md generated from the current project.
  • Game mode. The agent can drive our MCP tools to build a whole game: read the Pear kit reference, adapt a template, write the files, validate them, and run a simulated-player scenario, iterating up to 6 times until the scenario passes. File > New game from prompt streams this into Studio's Live Preview.
  • Simulated playtesters. We spin up one deep agent per player. Each plays like a person in the room: every turn it receives an Observation and replies with up to 5 actions (walk_to, look_at, gesture, give_device, say, wait) plus an unsure list of instructions it found unclear. Four personas (follows the rules, skims, cheats, confused) surface different problems. These agents test a game's written instructions, not its script behavior.

The runtime works with Anthropic, OpenAI or Google models. It also runs hosted next to our backend, where each account supplies its own key.

Pear Player is the iPhone/iPad app. It signs in, browses published games, and turns player movement into game inputs.

Hosting on Vultr. Pear's backend runs on a Vultr Ubuntu instance. A bootstrap script sets up Docker Compose with our Rust/Axum backend, Postgres and Caddy (TLS), and the backend also serves the built web Studio. That one box handles:

  • Game servers. A published game's server scripts run in an embedded JavaScript engine inside the backend, and players connect over WebSockets.
  • Accounts, publishing and discovery. Projects, versions and the game library live in Postgres, and the Player app reads from the same API.
  • Plugin distribution. Plugin bundles are stored as signed tarballs in Vultr Object Storage.
  • The hosted Deep Agents runtime, which the backend proxies to.
  • Deploys. A pull-based auto-deploy checks main every 60 seconds and rebuilds the box.

Stack: JavaScript (game scripts), Swift (Player), Rust + Axum + SQLx (backend), Python + LangChain Deep Agents (plugin), Postgres, Docker + Caddy on Vultr.

Challenges we ran into

Integrating Deep Agents into Studio. Deep Agents is a Python framework, and Studio is a desktop app, so the agent had to live as a plugin with its own runtime that Studio talks to over a websocket. That raised problems we hadn't planned for. We had to give the agent real write access to a project without letting it escape the project folder, so we scoped its permissions to what each plugin's manifest declares. We had to keep model API keys out of error messages. We had to support Anthropic, OpenAI and Google models, and we had to run the same runtime hosted next to our backend, with each account supplying its own key. Making the agents act like players, replying in a strict JSON action format that Studio could execute, took a lot of iteration.

Designing Studio's UI/UX and developer workflow. Nothing like Pear Studio exists to copy. We had to decide what it means to "develop" a game played with your body: how you place and configure virtual peripherals, how client and server scripts relate, how you test with simulated players, and how you go from editing to publishing. We reworked these flows several times before they felt coherent.

Making the whole system run together. Pear has many moving parts: the Player app, Studio, a simulator, a Rust backend, a script runtime, a plugin runtime, and Vultr infrastructure. Each had to agree on shared formats for device input, game files, remote events and accounts. A bug often turned out to live in a different system from where it showed up. For example, our Player app first opened every game to a blank 3D plane because 2D rendering wasn't supported, and that traced back to an early assumption in our project pipeline.

Accomplishments that we're proud of

  • Building Pear Studio, a custom desktop development environment for movement-based games.
  • Building an extensible plugin system and integrating LangChain Deep Agents to bring agentic AI into the development workflow.
  • Building simulated playtesters on LangChain Deep Agents that stress-test a game's instructions before real people do.
  • Developing Pear Player for iPhone and iPad to bring created games into the hands of players.
  • Deploying backend infrastructure, persistent project storage, and multiplayer game servers through Vultr.

What we learned

We learned that building a new entertainment platform requires intense planning and research. There was no existing tool for making games played with your body, so we had to decide what the workflow should even look like: how developers place peripherals, how client and server scripts relate, and how they test without owning every device.

We also explored how agentic AI can become an integral part of software development. Integrating LangChain Deep Agents through our plugin architecture taught us how AI agents can become extensible tools within an existing development environment rather than simply another conversational interface. Making that work meant giving the agent real context (a scripting skill and an AGENTS.md generated from the current project), real limits (permissions scoped to the project folder), and a strict output format that Studio could act on. We also learned that agents are useful for more than writing code: our simulated playtesters read a game's written instructions the way a person would and flag where they're unclear, which is a different kind of feedback than a test suite gives.

Finally, we gained experience designing a unified system that connects many different devices, simulated environments, developer tooling, and cloud-hosted services. A single standardized input format for real and simulated devices is what made that manageable, because games never had to care where an input came from. We also learned that in a system this interconnected, a bug often lives in a different part than where it shows up.

What's next for Pear

  • Expanded hardware support: Supporting additional Apple devices, including Vision Pro and third-party Bluetooth controllers.
  • AI-powered development: Expanding our Deep Agents integration to support more of the game creation, debugging, and optimization workflow.
  • Creator ecosystem: Expanding plugin capabilities, community publishing, moderation, and project version history.

Pear Banner

Built With

Share this project:

Updates

Submission history