Wayfinder

Inspiration

LLMs are a very popular topic in the tech space currently - a lot of projects are interested in using them, but we wanted to do something original and fresh with them. Actually using them to resolve rules of games, rather than as a front end output for something, allows for the player to enjoy endless possibilities, and make some really creative solutions to the problems.

What it does

The player is able to take photographs of objects and bring them as items into the game. They can then use these to interact with obstacles - for example, they could photograph a key, and then use that on a locked door to get through. There are also ways to transform items in game, such as an "anti machine" which turns the object into it's opposite (according to an LLM - so there's a bit of jank!)

How we built it

The project is structured in two parts: a backend written in Python using Flask, and a frontend mostly done with TypeScript. I'm not a fan of web frameworks so it's all that old-style document.getElementById stuff, but it works out well.

It's a single-player game, designed to be played on a mobile, but the server-client architecture works quite nicely and made the development reallly smooth. Basically, there's a global "game state" object which is kept in sync via HTTP between the frontend and the backend. In fact, we send the entire game state each time an action is taken, but it's gzipped so it's pretty fast.

The other main component is the LLM side of things. I have ollama running two models locally on my M1 MacBook, one for the text component and one for the visual reasoning. Surprisingly, the context switching between these doesn't slow things down too much? We're using phi4 (14B params) and llava.

Challenges we ran into

LLMs are very finicky and often ignore what you ask of them. We have to be really really specific in it's instructions in order for it to give as consistent and accurate a result as possible.