About the project
Oink Oink Lost is an AI-powered storybook adventure game where a little lost Piglet explores dynamically generated worlds, enters multiple rooms, talks with expressive NPCs, collects clues, and follows them to find Mama.
The project is AI-native from end to end. OpenAI models do not merely decorate a traditional game, they create the story, world, characters, rooms, dialogue, voices, clues, interactions, and final reunion. The entire application was also developed and deployed with OpenAI Codex, covering product ideation, frontend development, mobile-responsive gameplay, backend engineering, debugging, Docker packaging, and production operations.
Play it at oinkoink.fun or view the public repository.
Inspiration
Most AI story applications stop at generating text or a sequence of images. We wanted to explore a more ambitious question:
What if an AI-generated story became a world that a child could actually enter, explore, and influence?
That idea became Oink Oink Lost. Piglet has wandered away from Mama and must explore an illustrated world to find her. Instead of reading a fixed story, the player walks through the environment, enters rooms, speaks with characters, listens to their voices, finds clues, and solves a small mystery.
We also wanted to demonstrate the breadth of the OpenAI ecosystem. A single adventure combines structured reasoning, image generation, image understanding, interactive dialogue, speech generation, transcription, and moderation.
Finally, the development process itself became part of the experiment: could OpenAI Codex help take a game from an initial idea all the way to a polished, mobile-responsive, Dockerized production application? The answer was yes.
What it does
Players can choose a pre-generated adventure or describe a completely new setting, such as a moon station, a cloud toy shop, or a castle filled with friendly monsters.
For a new adventure, the game:
- Creates a structured story bible containing the setting, characters, rooms, clues, candidate locations, and final answer.
- Generates an illustrated hub world.
- Creates multiple enterable rooms matching that world.
- Generates Piglet’s character sprites.
- Uses visual understanding to identify doors, walkable areas, NPC positions, exits, and interactive hotspots.
- Creates three NPCs with distinct personalities, knowledge, voices, and conversational styles.
- Constructs a logically solvable three-clue mystery.
- Pre-generates common NPC interactions and audio to make conversations feel immediate.
- Lets the player talk naturally to NPCs using text, suggested dialogue choices, or a microphone.
- Generates a final reunion image when Piglet finds Mama.
The player can walk using a keyboard or tap directly on the world on a mobile device. NPC conversations are dynamic rather than fixed dialogue trees. Characters can answer questions about themselves and the world while remaining grounded in the current adventure.
For players who want a shorter experience, the game also offers a “too tired to find clues” path that automatically collects the clues but still lets the player complete the interactive reunion.
Completed adventures become reusable. Their generated assets and frequently used conversations are cached so future players can enter those worlds quickly without repeatedly paying the generation cost.
How we built it
Oink Oink Lost combines several OpenAI models and services into one generation and gameplay pipeline.
| Capability | OpenAI model | OpenAI service |
|---|---|---|
| Adventure bible, mystery, characters, rooms and structured story generation | gpt-5.4-mini-2026-03-17 |
Responses API |
| Hub hotspot detection, collision information and room annotation | gpt-5.4-mini-2026-03-17 |
Responses API with image input |
| Live, context-aware NPC conversations | gpt-5.6-luna |
Responses API |
| Hub, room, character, outline and reunion images | gpt-image-2 |
Images API for generation and editing |
| Narrator, NPC and Piglet voices | gpt-4o-mini-tts |
Audio Speech API |
| Player microphone recognition | gpt-4o-mini-transcribe |
Audio Transcriptions API |
| Screening custom adventure prompts | omni-moderation-latest |
Moderations API |
The frontend is built with React, TypeScript, Vite, Zustand, and PixiJS. PixiJS renders the explorable 2D world, directional Piglet sprites, movement, hotspots, room transitions, and animations.
The backend uses Python, FastAPI, Uvicorn, Pydantic, Pillow, and NumPy. Pydantic schemas keep generated stories structurally consistent. Pillow and NumPy process generated images and help derive collision and interaction data.
Adventure generation runs as a staged asynchronous pipeline. Independent tasks—such as room generation, character creation, NPC preparation, and audio generation—run concurrently to reduce waiting time. Progress is streamed to the browser using Server-Sent Events.
Generated adventures are stored as reusable JSON, PNG, and MP3 assets. Common NPC openings and the likely “Have you seen my Mama?” interaction are generated once and cached. Reunion images are also cached per source adventure, rather than regenerated for every play-through.
The application is packaged as a multi-stage Docker image and distributed through Docker Hub. A one-command installer lets anyone run it locally with Docker and their own OpenAI API key. Production runs on Google Cloud with Docker, Nginx, Caddy, automatic HTTPS, persistent storage, and the custom domain oinkoink.fun.
OpenAI Codex was used throughout the complete lifecycle:
- Product and gameplay brainstorming
- React and PixiJS development
- Responsive mobile UX
- FastAPI and OpenAI integration
- Prompt design and iteration
- Debugging visual, audio, and state-machine issues
- Voice experimentation
- Git and release management
- Docker packaging
- Google Cloud deployment
- HTTPS and reverse-proxy configuration
- Production diagnosis and fixes
Challenges we ran into
One of the hardest problems was turning a generated illustration into a playable environment. A visually beautiful world is not automatically a valid game map. We needed to identify walkable regions, doors, NPC locations, exits, and hotspots while keeping the artwork coherent. We solved this by combining image generation, an outline pass, vision-based annotation, and deterministic post-processing.
Another major challenge was ensuring that every generated mystery was actually solvable. Each clue needed to eliminate specific candidate locations without eliminating the correct answer. We added structured schemas, validation, retry instructions, and deterministic repair logic so the three clues converge on exactly one location.
AI-generated assets also finish at different times. Early versions allowed players to reach a room before its image was ready, or reach the ending while the reunion photo was still generating. We redesigned the generation pipeline, readiness rules, polling, retry behavior, and asset caching to make these transitions reliable.
Voice quality required substantial iteration. Generic speech sounded too adult and unnatural for Piglet. We tested different voices, performance instructions, and playback speeds before creating a tiny, cheerful baby-piglet voice. NPCs also received character-specific voices and speaking directions.
Maintaining visual consistency across a hub, multiple rooms, character sprites, and a reunion image was difficult. Prompt generation and strong shared visual direction helped each adventure feel like one coherent illustrated world instead of a collection of unrelated images.
Finally, making the game work across desktop and mobile required rethinking interaction rather than merely shrinking the interface. We added tap-to-move controls, responsive dialogue layouts, scalable game rendering, touch-friendly hotspots, and adaptive control legends.
Accomplishments that we're proud of
We are especially proud that Oink Oink Lost is not simply an AI chatbot wrapped in a game interface. The AI-generated output becomes a playable, spatial, multimodal world.
Some highlights include:
- Entire explorable adventures generated from a short user prompt
- Multiple visually consistent rooms within each world
- AI-generated mysteries with validated clue logic
- NPCs that support both guided choices and free-form conversation
- Distinct narrator, NPC, and Piglet voices
- Voice input through the player’s microphone
- Image understanding used to convert artwork into gameplay data
- Reusable adventures with cached images, audio, and common conversations
- A searchable global adventure library
- Desktop and mobile-responsive gameplay
- Downloadable and shareable reunion images
- One-command Docker installation
- A complete production deployment created and operated through Codex
Most importantly, the finished experience feels like an interactive storybook rather than a demonstration of disconnected AI APIs.
What we learned
We learned that AI-native products work best when models are given different responsibilities rather than asking one model to do everything.
Structured text generation is excellent for defining the world and its rules. Image generation brings that world to life. Vision turns the resulting artwork back into machine-readable gameplay information. Conversational models make characters interactive. TTS and transcription make the world feel alive and approachable.
We also learned that generative AI needs deterministic engineering around it. Schemas, validators, retries, caching, state machines, fallbacks, and post-processing were just as important as the prompts themselves.
Pre-generation and caching dramatically improve both responsiveness and cost. The best experience often comes from generating expensive content once, then reusing it intelligently while reserving live model calls for interactions that genuinely benefit from them.
Finally, OpenAI Codex changed how quickly we could iterate. We moved continuously between product design, code, visual debugging, voice experimentation, infrastructure, and production operations without needing separate toolchains or specialists for every domain.
What's next for OinkOinkLost
The next step is to evolve Oink Oink Lost from a hackathon project into a platform for reusable AI-generated story worlds.
- Longer adventures with connected chapters
- Recurring characters who remember previous encounters
- More puzzle and mini-game templates
- Themed background music and ambient soundscapes
- Parent- or teacher-created adventures
- Reading-level and language customization
- Cooperative adventures for families
- Personalized story collections
- Improved adventure discovery and community sharing
- Durable cloud storage and scalable generation workers
- Stronger safety, privacy, and age-appropriate interaction controls
Our long-term vision is a living storybook engine: describe a world, step inside it, talk with its characters, and turn imagination into a playable adventure using the full OpenAI ecosystem.
Built With
- caddy
- codex
- docker
- fastapi
- google-cloud
- gpt
- gpt-4o-mini-transcribe
- gpt-4o-mini-tts
- gpt-5.6-luna
- gpt-image-2
- omni-moderation-latest
- openai
- pixijs
- python
- react
- typescript
- vite
- zustand
Log in or sign up for Devpost to join the conversation.