-
List of available avatars
-
Rich customization options
-
How AI agent sees the game
Team: WHACKing my meat proxies
Inspiration
Existing AI companions on the market fail to deliver on truly interactive experiences. Often, they are uncanny, confidently incorrect, or are just too expensive for long gaming sessions. For them to feel like a true companion, a level of interactivity and proactiveness must be present, which AI agents built to simply follow instructions often lack.
We wanted to make something that can be its own person who is genuinely excited to learn more about you and what you’re playing. We wanted the model to pull accurate information to help you out when needed, and still be in character.
What it does
Purrtner accompanies you in your gaming session with relevant small talk, inquisitive questions, and proactive assistance. This is all possible with ~1s latency and costs only $0.02 per minute. The agent learns more on how to observe the game effectively with time and find out what relevant APIs are, and uses a versatile yet type-strict memory system to avoid context rot.
The player can use the companion to help with progress in-game, chat about the game and pull up wiki pages to aid with their experience. This increases accessibility and enjoyability, akin to gaming with a more experienced player helping you out.
How we built it
To outline our project, we utilised specialised systems for each major function:
- Speech transcription using Pipecat
- Cascading levels of complexity in computer vision ingested into Claude Haiku, with deep reasoning with Claude Sonnet
- Attention level mechanism to poll screenshots quicker during fast paced situations and conserve tokens in calmer moments
- Custom tool calls to populate AI context about the game: targeted area screenshots, looking at past events, automatic and persistent HUD recognition, displaying wiki articles, etc.
- A hybrid SQLite + markdown memory database, with additional SQLite Vector embeddings to improve long-term performance
- Voice generation with ElevenLabs With a frontend written in TypeScript on Electron and React.
Challenges we ran into
The largest challenge was ensuring the AI got consistent context, due to all the constraints we’ve put on cost. Sometimes the model would get confused about what was on screen, or directly contradicted what it said previously. This was due to a variety of reasons: the agent getting too in-character with its personality, having a 1,200 token limit on Gemini Flash (which led us to pivot to Claude), prompt engineering to encourage correct tool calls was incredibly difficult and required many iterations, etc.
Another challenge was to keep our token spending low. We introduced 3 systems to keep the price low without sacrificing quality.
Input-heavy calls are delegated to a cheaper model, which summarises them and feeds into the reasoning model. This includes regularly polled screenshots and HUD recognition.
Attention-level mechanism: the reasoning model infers moment intensity from multiple sources like screenshot summarization and HUD health values. During fast-paced gameplay like boss fights, the agent temporarily increases screenshot poll rate. It also has an option to look closer at any part of the screen, for example, when answering a user's questions.
We cache intensive computations. For instance, the model automatically recognizes a game’s HUD location and sets up a regular polling process. The HUD location is cached and automatically picked up on the next game launch.
We also struggled with the moral qualms of developing a tool that plays a part in the replacement of human-human interaction. We have made peace with this by agreeing that scientific advancements require sacrifices.
Accomplishments that we're proud of
Under our direction framework, our model can rapidly gain understanding in games with complex systems: testing on Terraria, Minecraft, and Slay the Spire - all non-trivial games - still yields great positive results.
Furthermore, the model adaptively switches from analysing 1 fps downscaled screenshots to a high-res, high-frequency feed in important events to save credits. We hit cache 97% of the time in our conversations.
Our memory system being designed based on real research papers tackling inconsistent memory helped us to guarantee quality. We maintained memory in 3 types: user-provided, observed, and researched from wiki. By typing, conflicting facts are weighed against each other, and proactive questions for confirming them get issued. At the same time, summaries of each session are written in .md, allowing for more complex behavior only when needed. (Sources: Context Rot: How Increasing Input Tokens Impacts LLM Performance | Chroma, [2504.19413] Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory, Effective context engineering for AI agents \ Anthropic.)
What we learned
With the abundance of available APIs and libraries for specific tasks, it was much more straightforward than we thought to build anything we wish. In this case, despite our lack of experience, we were able to carry out our plan by exploring all the tools given to us. Only by being willing to learn can we appropriately design the system that serves our specific function best.
Our research took us deep into understanding how to make LLMs work on complex tasks, especially in the face of real-time data! Problems involving incorporating multiple forms of context, that we took as unsolvable before, had ways of mitigating them. We just needed to take a deeper look.
What's next for Purrtner
Our persistent memory is great, but to ensure it works in long-term gaming spanning hundreds of hours, we’ll need more data. We can expand on existing tools to give the model more context and information sources.
We also want Purrtner to save your best moments, like a first boss kill or a narrow escape, into a gallery for each game you play. You can look back on them anytime, and Purrtner can sum up your achievements.
We also think this technology could have great applications in the real-world - for instance in smart glasses for people who are travelling. It can use the SQL memory database combined with computer vision to help the user identify new things in each country.
