21% of American adults (43 million people) are functionally illiterate (National Center for Education Statistics, 2019). That’s about 1 in 5 adults. And it’s only getting worse from here. This September, average reading scores for high school seniors fell to their lowest level in over 30 years (National Center for Education Statistics, 2024). Immediate action is necessary to ensure it is not here to stay. Models of success in improving literacy rates all share common themes: An emphasis on early reading education (grades K-3), strong fundamentals in oral language and vocabulary, and proactively monitoring student progress for early intervention (Harvard Gazette, 2025). At the same time, an additional challenge for instructors is to make reading compelling in the age of personal AI and media that offer the instant gratification that learning does not (Id.) — but games do. Multidimensional play is an essential part of education, promoting socio-emotional learning, cognitive development, and problem solving skills (White, 2012). In fact, research shows that learning is most effective when it occurs through interaction, guided participation, and leveraging peer-to-peer scaffolding (Chun & Cennamo, 2022).
Our team developed “Pirates,” an adaptive learning tool for students to improve fundamental vocabulary and critical thinking skills, while leveraging play and peer interaction. “Pirates” provides feedback to teachers about students’ vocabulary skills and progress, which is crucial for early intervention and monitoring of their education. To achieve these goals, we: (1) Used traditional computer vision techniques in combination with vision-language models to read the board state. (2) Efficiently process the board state to analyze all possible moves and create a list of adaptive learning strategies. (3) Provide a user-friendly interface that gives hints to students to encourage critical thinking and promote words at their level that are unfamiliar to them. (4) Created a real-time diagnostic on student progress and long-term analysis of their strategies and reading level.
Here’s a quick overview of how the word game underneath all of this works: imagine you’re playing banagrams, but for every tile you flip over, you also get the chance to make a word with someone else’s and steal the points for yourself (hence the name). This game is deceptively difficult. As we discovered, while analyzing possible moves from the board state, there are millions of combinations, and only a small portion are viable, which makes it challenging, but also incredibly rewarding, to solve without a computer.
At the core, we love playing games. Starting with a game that we love, we took a step to transform physical games into the digital world. We wanted to make tools that allow students to engage in activities they enjoy, while at the same time providing useful information for teachers to guide their journey, and improve America’s literacy rate.
So what was our biggest challenge? Certainly, the vision component of our pipeline. At first, we used off-the-shelf optical character recognition (OCR) models to parse the characters from the tiles. However, this method was incredibly inaccurate. Next, we attempted to train our own convolutional neural network (CNN) on tile recognition. With some initial promise, we created a data-pipeline to produce ~3000 labeled images for training. This showed significant accuracy in training, but with a small dataset, did not generalize well to other lighting conditions. We also tried to use a simpler system that would overlay characters on the image until it matches well; this had promise, but was outdone by using a vision language model (VLM). By building off of our initial attempts with a VLM, we were able to significantly improve the accuracy and consistency of our board state recognition. This journey was by far the most challenging of the hackathon, but we loved learning about many different computer vision processes along the way.
So what’s next? As it stands, Pirates is a powerful educational tool, but there are a few more steps to bring it to the world. First, we want to improve the vision system. While we only had one day to train a vision system, we expect that we could make the vision significantly more reliable in the future. Second, we hope to expand this type of tool to more games. This would provide more streams of data about student comprehension, enabling teachers with more effective insights. With these two paths, we see this tool taking strides to provide a fun, peer-interactive game that can enhance student experiences and education.
Log in or sign up for Devpost to join the conversation.