Threshold

An AI-directed adventure game where your real environment becomes the level, everyday objects become game pieces, your phone becomes a Lens, and Fire TVs become portals into the game.

Comment

Threshold: The Last Witness

Your home is the crime scene. Your television is watching. Your phone knows more than it should. And the Architect isn't necessarily telling the truth.

Inspiration

Most games ask players to enter a fictional world. I wanted to build one that enters theirs.

I started Threshold with a question: What if an AI-directed mystery could turn an ordinary home into a playable environment using the devices people already own?

Rather than building another game controlled entirely through a television or phone, I wanted to create an experience in which both devices were essential. Players would need to physically move through their surroundings, investigate real objects, communicate with one another, and make decisions that affected how the story unfolded.

The Fire TV would become a mysterious surveillance terminal. Each player's phone would become a private investigative tool called a Lens. Connecting them would be an AI-driven entity known as the Architect, whose instructions might be helpful, misleading, or deliberately manipulative.

That idea became Threshold: The Last Witness.

What it does

Threshold is an AI-directed, cross-device mystery game designed for 1–6 players, using 1–4 Fire TV devices and players' smartphones.

Each expedition begins by calibrating the environment. Players photograph areas of their home, and AI-assisted visual analysis helps the game identify physical objects and environmental details that can become part of the experience.

Once the expedition begins, the familiar space becomes something else.

Fire TVs operate as Thresholds: shared surveillance interfaces that display transmissions, mission information, interactive monitors, and instructions from the Architect.

Smartphones operate as Lenses: personal interfaces for examining evidence, receiving private information, making decisions, and interacting with the physical environment.

Players may be asked to investigate objects, notice changes in their surroundings, cooperate on challenges, or make decisions without knowing everything their teammates know.

The Architect directs the experience, but players cannot assume that every instruction is trustworthy.

The Last Witness

The current playable experience centers on a mystery called The Last Witness, with additional mysteries planned for future development.

Players investigate fragmented evidence while navigating an increasingly uncertain relationship with the Architect. Their discoveries, actions, and decisions influence how the expedition progresses toward its conclusion.

The experience supports solo investigation as well as multiplayer scenarios involving cooperation, competing information, and difficult choices.

Threshold is designed around an important principle: the physical environment is not just a backdrop. It is part of the game.

How I built it

Threshold combines native Fire TV development, a mobile web application, a shared multiplayer backend, AI-assisted environmental analysis, and AWS infrastructure.

Fire TV application

I developed the Fire TV application using Kotlin, Android, and Jetpack Compose.

The television serves as the shared interface for the expedition, displaying surveillance monitors, environmental information, Architect transmissions, and gameplay controls.

The application also manages device registration, session connectivity, remote-control interactions, and navigation between expedition phases.

Mobile Lens

I built the Lens using React and TypeScript.

The Lens provides each player with an individual interface for joining an expedition, capturing environmental evidence, investigating objects, receiving information, and participating in gameplay decisions.

Unlike a conventional second-screen companion, the Lens is integral to the experience.

Multiplayer backend

The session service uses Node.js, Express, and TypeScript.

It manages expedition sessions, registered Fire TVs, connected players, calibration data, gameplay progression, and shared state.

The backend coordinates actions across devices so that an interaction on one Lens or Fire TV can affect the shared expedition.

A structured game engine maintains the underlying state of the mystery while supporting AI-generated narrative and environmental interpretation.

AI integration

Threshold uses Google Gemini for AI-assisted calibration and environmental analysis.

My original plan was to use Amazon Bedrock with Amazon Nova Lite. However, despite completing the necessary account verification, my AWS account's applied inference quotas remained at zero. Rather than allowing that limitation to halt development, I integrated Gemini as an alternative.

Players' calibration photographs can be analyzed to identify objects and environmental details, allowing the game to incorporate elements of the actual playing space.

The AI integration uses a provider abstraction, preserving support for local development with Ollama and allowing future integrations without requiring major architectural changes.

AWS infrastructure

I deployed the session backend to Amazon EC2, running Amazon Linux 2023.

The infrastructure includes:

  • Amazon S3 for private calibration-image storage.
  • AWS IAM for scoped service permissions.
  • AWS Systems Manager for remote administration and secure configuration.
  • Amazon CloudWatch for structured operational logs and monitoring.
  • Caddy for HTTPS access to the backend.

Calibration images stored in S3 are subject to a one-day lifecycle expiration policy.

The architecture separates the Fire TV and Lens clients from the shared session service, enabling cross-device communication and centralized gameplay coordination.

Challenges I ran into

Building one game across multiple devices

One of the biggest engineering challenges was keeping multiple Fire TVs and phones synchronized.

A player could join or leave an expedition, a Fire TV could disconnect and reconnect, and the shared game needed to remain consistent.

This required careful session lifecycle management, device registration, reconnection handling, and backend-controlled state transitions.

Making the real world matter

Environmental gameplay introduced a different kind of challenge from traditional game development.

A conventional game designer knows exactly which objects exist in a level. Threshold must work with unfamiliar rooms, unpredictable lighting, different furniture, and environments it has never encountered.

I approached this by combining AI-assisted visual analysis with structured gameplay state. The goal was to make the environment meaningfully influence the experience without allowing generated narrative to override established game facts.

Integrating AI under real infrastructure constraints

I initially explored Amazon Bedrock for model inference, but account-specific inference quota restrictions prevented the intended integration.

Rather than allowing that dependency to block the project, I implemented Gemini through the existing AI provider abstraction while retaining local Ollama development support.

This reinforced the importance of designing integrations that can be replaced without rebuilding the entire application.

Developing and testing for Fire TV

Building for Fire TV introduced platform-specific challenges involving Android tooling, remote-control navigation, device networking, application lifecycle behavior, and testing across physical hardware.

I documented these issues and their resolutions in developer experience logs to preserve what I learned throughout the project.

Accomplishments I'm proud of

My biggest accomplishment is turning an ambitious concept into a functioning cross-device system.

Threshold brings together native television software, mobile interfaces, multiplayer state management, AI vision, cloud infrastructure, and physical-world gameplay.

I'm especially proud of:

  • Creating a game where the television and phone have distinct, necessary roles.
  • Connecting AI-assisted environmental understanding to actual gameplay.
  • Building a multiplayer architecture that supports multiple Fire TVs and phones.
  • Deploying the backend to AWS with private image storage and operational monitoring.
  • Developing the project independently while working through unfamiliar technical challenges.

Most importantly, Threshold demonstrates that a game doesn't need a predetermined virtual map to create an interactive world. The player's surroundings can become the level.

What I learned

Threshold pushed me to think beyond individual features and focus on how an entire system behaves.

I gained practical experience with native Android development, multi-device communication, cloud deployment, AI provider integration, session persistence, and operational observability.

I also learned how important it is to balance experimentation with deterministic behavior. AI can make a game surprising, but a reliable gameplay engine must still maintain consistent rules, evidence, and consequences.

The project strengthened my understanding of software architecture, debugging across system boundaries, and building an experience around the strengths of different devices.

What's next for Threshold

Threshold: The Last Witness is the foundation for a much larger idea.

Future development will include additional mysteries, richer environmental mapping, expanded Architect behavior, more sophisticated procedural encounters, and improved multiplayer interactions.

One of the most important decisions I faced during development was choosing between the full 15–20-minute experience I originally envisioned and a more focused, polished 8–10-minute playable expedition. Given the hackathon's time constraints, I chose the latter. I wanted to prioritize a cohesive, interactive experience rather than attempt to deliver every planned feature at the expense of quality.

After judging, I plan to revisit Threshold and continue developing it toward the experience I originally imagined. My goal is for players to finish an expedition thinking, "Holy crap, we did all that with just a few pictures of my house?"

I also plan to implement stricter privacy policies, more comprehensive database lifecycle management, and additional safeguards around environmental data. These improvements will be important as Threshold evolves beyond a hackathon project.

Ultimately, I want Threshold to become a game people choose to play whenever they have company over—something that makes an ordinary gathering unexpectedly memorable.

I also want to continue refining the onboarding experience, simplifying how players connect their Lenses, and preparing the Fire TV application for broader distribution.

Another idea I'm particularly interested in exploring is Ring camera integration. While players' phones would continue serving as Lenses, Ring cameras could become the Eyes, extending the game's surveillance system into the physical environment. This could introduce live environmental observations and entirely new gameplay possibilities, making the experience feel even more immersive.

The long-term vision is a platform for AI-directed experiences that transform ordinary spaces into interactive worlds.

Threshold begins when the screen stops being the boundary of the game.

Built With

Share this project:

Updates

Submission history