-
Landing Page
-
Playground Page
-
Game Selection Screen
-
Game Rules
-
Gameplay
-
Stats Page
-
Stats Page Live Data
-
Global Data Visualization
-
Spatial Death Heatmap marking where players frequently die in the game
-
Component Catalogue
-
-
Inspiration
When we were younger, ordinary objects could become part of completely different worlds. A stack of books could become a staircase, a charging cable could become an obstacle, and the space between two desks could become an entire platforming level.
We wanted to recreate that feeling using computer vision and game development.
Most game-creation tools still require people to write prompts, learn an editor, design a level, or understand game-development concepts. With Playground, the only thing a user needs to do is take a photo.
The idea was simple: what if the objects around you could become a playable arcade game?
What it does
Playground turns a photo of everyday objects into a personalized 2D arcade game.
A user can take a picture of their desk, bedroom, kitchen counter, backpack, or almost any group of objects. Our computer vision model analyzes the image and identifies the objects inside it, along with their positions, sizes, shapes, and physical properties.
Based on the image, the system calculates the best game to generate and determines which gameplay features should be included.
For example:
- Books can become platforms.
- Small objects can become collectibles or ammunition.
- Appliances can become machines that launch projectiles.
- Soft objects can become bounce pads.
- Sharp objects can become hazards.
- Large objects can become falling obstacles.
- A central object can become the main enemy, goal, or environmental feature.
The image does not only change the appearance of the game. It influences the game type, objective, difficulty, mechanics, hazards, collectibles, enemies, projectiles, level structure, and win conditions.
Depending on what is detected, Playground can generate several different game experiences
Playground also includes a live analytics dashboard that displays information from real gameplay sessions, including active players, deaths, completion rates, detected objects, mechanic usage, and death locations.
No prompt, level editor, game-development experience, account, or mobile application is required.
How we built it
System architecture
┌─────────────────────────────────────────────────────────┐
│ Frontend (Next.js 15) │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Photo & QR│ │React │ │TypeScript│ │Tailwind │ │
│ │Capture │ │Components│ │ │ │CSS │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ API Layer (Next.js API Routes) │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Upload & │ │Game │ │Publishing│ │Telemetry │ │
│ │Capture │ │Generation│ │& Sharing │ │Events │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Image Processing & Vision Layer │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Sharp │ │Computer │ │Object │ │Zod Data │ │
│ │Processing│ │Vision │ │Detection │ │Validation│ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Game Intelligence & Selection Layer │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Game Type │ │Object Role│ │Difficulty│ │Feature │ │
│ │Selection │ │Assignment │ │Calculator│ │Selection │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Game Generation Pipeline │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Level │ │Mechanic │ │Game Spec │ │Safety │ │
│ │Structure │ │Selection │ │Builder │ │Validator │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Game Engine (Phaser 3) │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Physics & │ │Player │ │Hazards & │ │Goals & │ │
│ │Movement │ │Controls │ │Enemies │ │Scoring │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────┘
│
┌─────────┴─────────┐
▼ ▼
┌──────────────────────┐ ┌──────────────────────┐
│ Database Layer │ │ Storage Layer │
│ ┌──────────────────┐│ │ ┌────────────────┐ │
│ │Supabase Postgres ││ │ │Supabase Storage│ │
│ │- Game Data ││ │ │- Source Images │ │
│ │- Player Sessions ││ │ │- Game Assets │ │
│ │- Event Telemetry ││ │ │- Capture Files │ │
│ │- Leaderboards ││ │ │- Thumbnails │ │
│ └──────────────────┘│ │ └────────────────┘ │
└──────────────────────┘ └──────────────────────┘
│ │
└─────────┬─────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ Publishing & Community Layer │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Public │ │Shareable │ │Likes & │ │Global │ │
│ │Playground│ │Game Links│ │Remixes │ │Rankings │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Live Analytics Dashboard │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Active │ │Death │ │Completion│ │Mechanic │ │
│ │Sessions │ │Heatmaps │ │Analytics │ │Usage │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────┘
Cross-device photo capture
On mobile devices, Playground can open the camera directly.
On desktop, users can upload an image or generate a one-time QR code. Scanning the code opens a camera page on their phone. Once the user takes a photo, it automatically appears inside the original desktop session.
To make this work securely, we created a custom capture-session system with:
- Separate phone and desktop tokens
- Hashed token storage
- Ten-minute expiration periods
- Single-use image claims
- Polling for capture status
- Duplicate-upload protection
- Recovery for interrupted uploads
- Automatic cleanup of expired sessions
The experience does not require an account, mobile application, or push notifications.
Image processing
Every uploaded photo passes through the same server-side image pipeline.
We use Sharp to:
- Validate the uploaded file.
- Decode the image.
- Correct its orientation.
- Remove inconsistent metadata.
- Convert it into a normalized JPEG.
- Store the normalized image.
This prevents device-specific image formats, browser differences, and camera metadata from producing inconsistent results.
Computer vision
After normalization, the image is sent through our computer vision pipeline.
The model returns structured information about the detected objects, including:
- Object labels
- Positions
- Dimensions
- Confidence scores
- Physical properties
- Possible gameplay roles
- Uncertain detections
- Relationships between nearby objects
The model’s response must match a strict JSON structure. We validate the response using Zod before allowing it to enter the game-generation pipeline.
If the response is incomplete or incorrectly formatted, the system retries the analysis using the validation issues as feedback.
Game intelligence
Once the image has been analyzed, Playground calculates which game experience fits the detected objects best.
The system considers:
- The number of objects
- Their relative positions
- Their sizes and shapes
- Their physical properties
- Whether the image contains flat surfaces
- Whether it contains small throwable objects
- Whether it contains machine-like objects
- Whether objects can become hazards
- Whether objects can become collectibles
- The expected difficulty of the generated level
From this information, the system calculates:
- The game type
- The main objective
- The player’s goal
- The level structure
- The difficulty
- The required mechanics
- Object roles
- Enemy behaviour
- Hazard placement
- Collectible placement
- Projectile behaviour
- Time limits
- Victory conditions
For example, a large appliance surrounded by smaller items may produce a shooter game where the appliance launches those objects.
A photo containing many large items may produce a survival game where those items fall from the sky.
A photo with several small objects may produce a timed collection game.
A photo containing long or flat objects may produce a platforming-focused experience.
Object-role assignment
Playground separates what an object looks like from how it behaves inside the game.
Each detected object can be assigned a gameplay role such as:
- Platform
- Moving platform
- Hazard
- Collectible
- Projectile
- Enemy
- Target
- Checkpoint
- Portal
- Goal
The system then matches the object with supported game artwork. When an exact object match is available, the game uses artwork representing that object. When an exact match is unavailable, Playground selects a compatible visual while preserving the intended mechanic.
This lets the photo meaningfully influence the game without allowing unpredictable image-analysis results to directly control the physics engine.
Game specification and safety validation
After deciding what the game should contain, Playground creates a structured game specification.
The specification defines:
- The player
- World dimensions
- Platforms
- Moving entities
- Collectibles
- Hazards
- Projectiles
- Enemies
- Checkpoints
- Portals
- Timers
- Scoring
- Win conditions
Before the game can be loaded, it passes through a deterministic safety validator.
The validator checks:
- Entity boundaries
- Player spawn position
- Platform availability
- Reachability
- Safe landing areas
- Goal and portal counts
- Collectible reachability
- Moving-platform speed and distance
- Projectile fairness
- Hazard placement
- Timer requirements
- Mode-specific rules
- Mechanic and visual compatibility
- Victory-condition validity
If the generated configuration is impossible, unfair, or invalid, it is rejected and recalculated before the user sees it.
Computer vision decides what the image can become, while deterministic code ensures the final result is playable.
Game engine
We used Phaser 3 to render and run the generated games.
Phaser handles:
- Physics
- Collision detection
- Player movement
- Jumping
- Mobile controls
- Keyboard controls
- Moving platforms
- Bounce pads
- Hazards
- Enemies
- Projectiles
- Collectibles
- Checkpoints
- Portals
- Falling objects
- Timers
- Restarts
- Scoring
- Victory conditions
Gameplay events are sent through a shared event bus. This lets the Phaser game communicate with the React interface and analytics system without tightly connecting the game engine to the rest of the application.
Publishing and community
Once a game is generated, users can publish it to the public Playground.
Publishing creates:
- A unique game page
- A shareable link
- A leaderboard
- Completion statistics
- Death statistics
- Like counts
- Remix counts
- Detected object information
Games can be discovered and sorted by categories such as newest, trending, hardest, fastest completion, and most remixed.
Live analytics
We connected telemetry directly to the game engine.
The system records events such as:
- Game started
- Player died
- Game restarted
- Collectible collected
- Checkpoint reached
- Target destroyed
- Game completed
- Game published
- Game remixed
- Mechanic discovered
The analytics dashboard displays:
- Total objects scanned
- Published games
- Games played
- Completed runs
- Total deaths
- Average completion rate
- Most detected objects
- Mechanic usage
- Games generated over time
- Difficulty compared with completion rate
- Recent community activity
- Active sessions
- Fatality counts
- Spatial death heatmaps
- Live gameplay events
- Object relationships within a selected game
The dashboard checks for new telemetry every 2.5 seconds.
Death events include the player’s coordinates, allowing Playground to display death locations directly over the level structure instead of only showing a total number.
Challenges we ran into
Deciding which game best fits an image
Every image can contain a completely different combination of objects.
One user might photograph a laptop, books, and an eraser. Another might photograph a mug, fruit, and a microwave.
We needed the system to do more than replace game artwork. It had to understand which mechanics each object could support and calculate a complete game around them.
We solved this by first converting the image into structured object data. The system then uses that information to determine the objective, game type, difficulty, mechanics, and object roles.
Keeping generated games playable
A computer vision model can identify an object, but it cannot guarantee that a jump is possible, a collectible is reachable, or a projectile is fair.
We separated the generation process into three responsibilities:
- Computer vision understands the image.
- The game-intelligence system calculates the best experience.
- Deterministic validation guarantees that the result is playable.
This separation allowed us to use AI for personalization without allowing unpredictable output to directly control game physics.
Building the QR capture flow
We wanted users creating on a desktop to take a photo using their phone without manually transferring the file.
Creating this experience without accounts or a mobile application required us to handle:
- Session ownership
- Expiring tokens
- Upload concurrency
- Polling
- Cancellation
- Duplicate uploads
- Interrupted uploads
- Token security
- Automatic desktop handoff
Although the final interaction feels simple, it functions like a small distributed system.
Handling different environments
One of our capture-session tests expected the application to use local in-memory image storage.
The test passed locally because our machines did not have production Supabase credentials. During Vercel’s build process, the same test inherited production environment variables and switched to real Supabase Storage.
The assertion failed, and the test also uploaded an image to production storage.
We fixed this by explicitly removing Supabase credentials during the test and restoring them afterward. This made the test behave consistently across local development, CI, and cloud deployment.
Making analytics real
It would have been easy to create an impressive-looking dashboard using static or simulated information.
Instead, we connected analytics directly to real game events. Every death, restart, collectible, checkpoint, completion, destroyed target, and published game enters the same event pipeline used by the dashboard.
When there is no data, the dashboard displays an empty state instead of showing fabricated numbers.
Accomplishments that we're proud of
Building the full photo-to-game experience
A user can take a photo, generate a game, play it, publish it, share it, compete on its leaderboard, and view its gameplay analytics without leaving the platform.
Making images affect the actual game
The uploaded photo does not only change colours or background artwork.
Detected objects influence:
- The game type
- Objective
- Difficulty
- Level structure
- Platforms
- Collectibles
- Hazards
- Enemies
- Projectiles
- Artwork
- Rules
- Win conditions
Supporting multiple game experiences
The same image-analysis pipeline can create platforming games, shooters, survival games, timed collection challenges, and projectile-dodging courses.
Creating secure cross-device capture
Our phone-to-desktop capture experience works without requiring users to create an account or install an application. Capture tokens expire, raw tokens are not stored, and each uploaded image can only be claimed once.
Adding a deterministic safety layer
Every game is validated before it reaches Phaser. Reachability, boundaries, landing areas, hazards, projectiles, moving entities, collectibles, timers, and victory conditions all have rules enforced in code.
Building real gameplay intelligence
The analytics dashboard displays real active sessions, fatalities, death locations, completion data, mechanic usage, and detected-object statistics from actual games.
Creating a complete community layer
Generated games are not temporary demos. Users can publish them, share them, like them, remix them, discover other games, and compete on leaderboards.
What we learned
AI works better with a focused responsibility
Our computer vision model is responsible for understanding the image and identifying what the detected objects could become. Deterministic application code remains responsible for physics, fairness, validation, and completion. Keeping those responsibilities separate made the system significantly more reliable.
Personalization needs to change gameplay
Simply replacing a platform sprite with a book would not have created a meaningful connection to the photo. The image needed to influence the game’s objective, mechanics, difficulty, hazards, collectibles, and rules. Using structured properties from each detected object helped us make the final game feel connected to the original image.
Generated content still needs deterministic rules
AI can make creative decisions, but game physics require predictable constraints. By validating every game specification before loading it, we could reject impossible configurations instead of hoping that the generated result worked.
What's next for Playground
Multiplayer experiences
We want to explore cooperative and competitive games where multiple players interact with the same generated environment.
Improved computer vision
We want to improve object segmentation, depth estimation, object relationships, material detection, and confidence handling. This would allow Playground to make more detailed decisions about how objects should behave.
Smarter gameplay analytics
We want to use player behaviour to identify levels that are too difficult, unfair, or confusing.
Built With
- motion
- next.js
- phaser.js
- playwright
- supabase
- tailwind
- typescript
- vercel
- vitest
Log in or sign up for Devpost to join the conversation.