Inspiration
Games usually react to buttons, not to the person holding the controller. We wanted to explore what happens when a game can interpret live physiological signals and adapt its atmosphere while remaining fair, playable, and fun.
What it does
EchoShift is a futuristic 2D platformer that responds to the player’s interpreted state. Camera-based Presage measurements pass through our deterministic Player State Engine, which produces four gameplay states: CALM, ENGAGED, HIGH_AROUSAL, and UNKNOWN.
The environment and selected gameplay behaviors transition smoothly between modes. A Gemini-powered companion uses the current game context to provide relevant dialogue, while ElevenLabs gives the companion a voice. A clearly labelled Demo Mode keeps the experience playable when live sensing is unavailable.
These states are gameplay interpretations—not medical diagnoses or claims about the player’s emotions.
How we built it
We built the frontend with React, TypeScript, and vite. Presage measurements are normalized into a shared PlayerMetrics format before entering our Player State Engine.
To prevent rapid mode switching, the engine smooths readings, applies hysteresis, confirms candidate states, and keeps normal states active for a minimum duration. Phaser then gradually interpolates colors, lighting, environmental activity, and bounded gameplay parameters instead of instantly switching the entire scene.
Gemini receives structured game context and returns bounded dialogue and events. ElevenLabs converts the resulting dialogue into speech, with text and predefined fallbacks available if either service fails.
Challenges we ran into
The largest challenge was turning noisy, delayed, or missing physiological measurements into stable gameplay behavior. We needed to distinguish brief fluctuations from meaningful changes without making the game feel unresponsive.
We also had to keep transitions fair. A state change should never suddenly accelerate a platform, teleport an enemy, or cause unavoidable damage. Finally, we designed fallbacks so camera, AI, or voice failures would never stop the core game.
Accomplishments that we're proud of
We created a clean separation between sensing, state interpretation, gameplay, AI dialogue, and voice. Our deterministic state engine prevents flickering, handles missing data explicitly, and keeps raw measurements from directly controlling the game.
We are also proud that EchoShift remains understandable and playable in every state, including UNKNOWN, and that simulated input is clearly identified as Demo Mode.
What we learned
We learned that adaptive experiences need stability as much as responsiveness. Smooth presentation, minimum state durations, hysteresis, and safe gameplay boundaries are essential when real-world signals influence a game.
We also learned that generative AI works best as a bounded creative layer: the game supplies verified context and allowed actions, while deterministic code retains control of gameplay.
What's next for EchoShift
Next, we want to tune the state engine through more playtesting, expand the futuristic level, improve accessibility, and add more context-aware companion interactions. We would also explore additional Presage signals and adaptive mechanics while keeping the experience transparent, fair, and non-medical.
Log in or sign up for Devpost to join the conversation.