Inspiration
What if the objects sitting on your desk could become the setting of your next gaming adventure? We wanted to bridge the physical and digital worlds by turning an ordinary workspace into a unique, playable platformer. Inspired by the creativity of game generation and the familiarity of everyday objects, we created Deskcapades.
Most games ask you to leave your surroundings behind. We wanted the opposite and allow the user to point a camera at their workspace, one that you already know. By doing this, we let the familiar become the foundation for something new. The layout of your actual desk generates a new level no one else has played before.
We were inspired by the idea of generative games, where the systems surprise you with unexpected spaces, and by the creativity of objects that are already present around you. Instead of designing levels from scratch, we are treating the real world as a collaborator. The result? A platformer that you have a personal connection with
What it does
Deskcapades turns your desk into a pixel-art platformer. Point your laptop webcam or phone camera at your desk, and everyday objects become parts of a playable garden-castle world. A pencil might become a wooden bridge, a can a climbable tower, a mug a swimming pool, and a computer mouse a trampoline. The pieces appear where the real objects are, using their positions, sizes, and colors.
Rearrange your desk, and you redesign the level. Each object type has a consistent transformation, and AI helps match unfamiliar objects to an existing game piece.
Two game modes
- Story mode: Play on a level built from your live desk camera. Move objects around to change the level, collect floating seeds, and watch the garden bloom. You can freeze a setup whenever you want to keep playing the same level.
- Challenge mode: Take a photo of your desk and let an AI level designer build an obstacle course around it. Choose from four difficulties, collect every coin, avoid hazards, and reach the flag before time runs out. Your best times and points are saved.
The world is designed to feel like a platformer rather than stickers on a photo. Raised objects sit on floating grassy islands, and the pieces have their own behaviors: climb towers, swim in pools, bounce on mushrooms, slide on ice, or avoid spikes and lava.
You can also play as one of 10 pixel characters or create a custom character from a photo of a toy or stuffed animal. If your phone is easier to position above your desk, pair it with the laptop using a QR code and use it as a wireless camera and remote.
How we built it
Deskcapades is built in TypeScript in a single repository with three packages: a browser client, a Node.js server, and shared types for communication between them. The client uses Vite and HTML5 Canvas. Instead of using a game engine, we built the platforming physics, collisions, rendering, and pixel-art drawing ourselves.
From a real desk to a playable level
The camera feed is sent to the server, where Perceptron detects objects and returns their labels and positions. We filter out irrelevant detections, merge duplicate boxes, and sample each object's colors. The game then maps each object to a piece using a fixed set of rules, with Google Gemini helping match objects that are not covered by those rules.
In Story mode, scans arrive over time, so we match new detections to objects already on screen. This keeps pieces moving smoothly instead of making them flicker whenever an object is briefly missed.
Each piece is translated into game geometry, such as platforms, climbable areas, hazards, or special effects. The world and character scale with the detected object sizes, helping the game remain playable when the camera moves farther away.
AI-generated challenge courses
For Challenge mode, Gemini designs a course using the detected desk objects, the selected difficulty, and the player's movement limits. We don't rely on the AI output without checking it. The game validates the course, adjusts placements, and adds reachable platforms when needed. If hazards make a course impossible, they can be removed, and a built-in course designer acts as a fallback if the AI fails.
We also wrote automated tests that generate hundreds of randomized desk layouts across difficulty levels and camera scales to check that the goal remains reachable. This was an important part of making AI-generated levels challenging without making them unwinnable.
A custom game engine and pixel art
The game loop handles movement, jumping, climbing, swimming, bouncing, sliding, hazards, collectibles, and challenge objectives. Every platform and character is drawn in code on a pixel grid, with colors sampled from the real objects on the desk. Sound effects and looping chiptune music are generated with the Web Audio API, so there are no pre-recorded audio files.
Connecting the phone and laptop
A phone can stream camera frames to the laptop through a WebSocket connection, and the same connection carries controls such as Snap and Freeze. Each laptop tab gets its own private room, so different players can use the game at the same time without sharing their desk feeds.
The app is hosted on Render over HTTPS, which allows browsers to access the camera. The server handles the game API and phone connections, while API keys are stored in deployment environment settings rather than in the source code.
Building it as a team
We built Deskcapades in stages: connecting the camera, detecting objects, turning them into game pieces, drawing the pixel-art world, and adding gameplay. We playtested with real desk objects and tested key mechanics such as ladders, rivers, ice, hazards, and course reachability.
Challenges we ran into
One of our biggest challenges was translating real-world objects into a functional platforming environment. Objects come in different shapes, sizes, and positions, so turning a desk into a level that feels natural and playable requires careful mapping and level generation. We also had to balance procedural variety with reliable gameplay, ensuring that obstacles provide a challenge without making levels impossible to complete.
What we learned
Developing Deskcapades taught us how computer vision, game physics, and procedural generation can work together to create interactive experiences from real-world environments. The computer vision aspect introduced variability within our platform, as the different labels and bounding boxes might be imprecise. Generating levels from real-world environments also highlighted the differences between workspaces. Thus, layouts that function effectively on an uncluttered surface produced unplayable gaps when additional objects were present.
Through this experience, we also learned the importance of designing around technical constraints, testing generated levels, and balancing creativity with functionality. Deliberate decisions regarding platform dimensions, collision response, and detection thresholds were necessary to maintain playability even when the vision output was inconsistent. Balancing creative ambition with practical limitations remained at the forefront of our platform and design.
Future Steps
Our immediate priorities are to improve object detection accuracy and increase the reliability of level generation across diverse desk configurations. Certain environments currently produce engaging layouts, while others yield sparse or difficult levels. We aim to strengthen the system’s ability to interpret ordinary, cluttered workspaces so that players are not required to rearrange their surroundings in order to obtain a playable experience.
Gameplay refinement is equally important. A generated level may appear visually compelling yet prove unsatisfying to play, or remain technically completable while feeling frustrating. Additional iteration will focus on ensuring that most levels provide a clear path forward and remain enjoyable.
Looking ahead, we intend to explore multiplayer functionality and further polish the overall user experience. The objective is to advance Deskcapades from a functional prototype toward a more robust, accessible, and consistent product that remains engaging even under imperfect camera or detection conditions.
Built With
- gemini
- javascript
- node.js
- perceptron
- typescript
