AIPictionary

Inspiration

We wanted to make Pictionary more immersive by removing the need for a physical drawing interface. Instead of picking up a mouse, using a trackpad, or touching a screen, we wanted players to simply use their hands to draw.

That idea became the foundation of our project: a Pictionary game where a webcam watches your hand movements and turns them into a digital drawing. You can draw, pause, and clear the canvas using gestures, without physically interacting with the computer.

To complete the experience, we added an AI model that looks at the finished drawing and tries to guess what you drew.

How We Built It

We built the project in Python using OpenCV and MediaPipe, with an AI model handling the drawing recognition.

The core pipeline is:

Webcam → Hand Tracking → Gesture Recognition → Digital Canvas → AI Guess

MediaPipe detects hand landmarks from the webcam feed. We use those landmarks to recognize gestures and track the position of the player's finger.

OpenCV takes that movement and renders it onto a virtual canvas in real time. Different gestures allow the player to draw, erase, pause, and clear the canvas.

Once the player finishes their drawing, we send it to an AI model (GPT 5.6-Sol), which analyzes the image and attempts to guess what was drawn.

What We Learned

We learned how much goes into making a computer vision interaction feel natural. Tracking a hand is one thing; turning that tracking data into intuitive controls is another.

We experimented with different gesture definitions and detection thresholds to make drawing feel responsive while preventing accidental erasing or pausing.

We also learned how computer vision and AI can complement each other: computer vision gives the player a natural way to interact with the game, while the AI provides the intelligence needed to interpret their drawing.

Challenges We Faced

One of our biggest challenges was getting an AI model to reliably guess our drawings. Pictionary sketches can be rough, abstract, and very different from the images an AI model is typically trained to recognize, so getting accurate guesses required experimentation.

We also initially wanted to build a multiplayer experience with an appealing frontend UI, but connecting multiple players through a network while keeping the game state synchronized proved difficult within the time constraints of the hackathon.

Another challenge was creating a web-based application that could communicate effectively with our Python backend. We experimented with connecting the frontend and backend, but getting the full system working reliably was more involved than we anticipated.

Although we weren't able to complete the full multiplayer web experience, these challenges helped us understand the complexity of connecting computer vision, AI, networking, and frontend development into one cohesive application. We ultimately focused on making the core experience work: drawing with our hands and having an AI attempt to understand what we created.