Touch Grass

The home dashboard for logged in users.
The game display for the first game.
The video display for the second game
The graphics during the user response during the second game
The final graphics after the second game
The join screen for hosting the game.

Inspiration

Our inspiration was the studio “Jackbox Games,” known for games such as Quiplash or Murder Mystery. We took their premise of one computer displaying the game screen and people playing on their phones, and extended it further, really forcing people to get up, be quick on their feet, fast with their writing, and maybe touch some grass in the process.

What it does

Our game is a website that users can access on their laptop and create their own account in order to host the games. Once they have started hosting a game, people can pull up their phone to the same website, which will redirect them to a play screen. Up to four players can put in a four character code to connect to the game, then the host can start the game. After a brief explanation and an animation, the program uses an LLM to generate context-specific objects for the players to find. Then, players rush to take photos of those objects (images are analyzed by Gemini) and upload them to the server. Players who find items first score more points. which they are analyzed by Gemini to identify the objects within those images. In the second round, an Instagram reel plays and players have to answer a quiz after the reel has played to test if their attention was focused. Gemini autogrades their responses, and adds the sum of the grade to their total score. Whichever player has the highest score at the end of the game wins.

How we built it

We combined plenty of different systems in our application. At the design phase, we used Figma to lay out how our application will look like. To start off the code, the main backend is built with Node.js, using Express.js for the main network routing. This backend is connected to a MongoDB cluster which hosts the account data. This is reverse-proxied through the frontend, which is a Next.js/React frontend. The main game is built via state machines and various React components, allowing for easy expansion in the future. The frontend communicates with the backend via REST API for general requests for things such as starting a game or querying states. In the live game, the communication protocol is switched over to a websocket to allow for fast and easy communication, and little overhead for event timings. The individual players also communicate with the backend via a websocket. For various requests, when we require use of an LLM - take the example of analyzing an image - we host a second backend server that is only exposed to the Express backend, built with Flask and Google’s Gemini, which can accept inputs for images and text, allowing us to analyze the images for data. We originally had a YOLO11 model running in Python for image recognition, but we found that Gemini was far superior to the object detection model. The voicelines were generated with Elevenlabs, and images were found from Google.

Challenges we ran into

We were a small team, only two people! One of us was unavailable for half the competition, so we had to work extra hard in order to get this app off the ground. Despite this challenge, we were able to be successful and create a fun game! Furthermore, originally, we tried to use a YOLO11 model for object detection for the scavenger hunt, but we found that Gemini was far more specific and accurate in its object detection than the YOLO11 model.

Accomplishments that we're proud of

Given the challenge of integrating many systems together, such as [Next.JS](http://Next.JS), Gemini AI models, Flask, and Express JS, we’re really proud that despite our challenges that we mentioned above, we were able to finish our game in time \- and have fun playing it\!

What we learned

We learned how to use Gemini as an effective image recognition model, as well as an effective auto-grading algorithm, and how to access these usages through running a Flask server and learning how to extensively prompt the AI for specific tasks.

What's next for Touch Grass

Our main future directions for Touch Grass is to expand the difficulty to make it more enjoyable with a larger number of players. For example, the difficulty would increase over multiple rounds (less time for search and simultaneous reels) and eliminate one player per round.  Another direction could be to integrate Instagram into the social aspect of this project by allowing users to pick reels of their choice, of which some would be randomly selected.