Chat2Craft

Inspiration

Minecraft is widely considered one of the most culturally significant and best-selling games of all time, giving players the freedom to explore, create, survive, and connect with others. However, not everyone can fully experience that freedom. For players with physical disabilities, like amputated limbs or limited motor control, traditional keyboard and mouse inputs can make even basic actions like moving, turning, switching items, placing blocks, or interacting with the world difficult or exhausting. This creates a real accessibility barrier. Minecraft is a game built around being creative and open-ended, yet we found that play can become hard to access simply because the controls are not equally usable for everyone.

What it does

For this hackathon, we built Chat2Craft for the Health & Wellbeing track because accessibility is a major part of wellbeing and healthcare. As shown by the study “Benefits, Barriers, and Accessibility in Video Games: A Focus Group Study of College Students with Disabilities,” gaming can support social connection, stress relief, and self-expression, but many disabled players face barriers that prevent them from even getting these experiences. Specialized adaptive controllers and assistive gaming setups can be expensive, hard to configure, or unavailable to people who just want to boot up a game and play. For example, the Xbox Adaptive Controller has a base price of $100. Most people, however, need to spend upwards of more than $300 extra to pay for the accessories to meet their specific needs. Additionally, third-party options like the Hori Flex controller have a base price of $250. Our goal was to lower this barrier by creating a Minecraft accessibility tool that uses voice and natural language to help players control the game with less reliance on traditional keyboard and mouse inputs.

Minecraft can require constant movement, camera control, item switching, and actions like placing or breaking blocks. For some players, those inputs can be tiring, painful, or inaccessible. Our mod lets players use simple spoken commands like “walk forward,” “turn left,” “stop,” or “switch to slot 8,” to lower the accessibility barrier to a simple verbal command.

How we built it

We used Claude to help translate natural language into Minecraft actions. Instead of requiring the player to memorize exact commands, the player can speak more naturally, and the AI helps interpret the intent into an action. We also used AI during development to brainstorm accessibility boundaries, design error handling, and think through how to keep the tool helpful without making it unfair.

For the Minecraft control layer, we used MacroKeybind. MacroKeybind is a Minecraft mod that lets players bind actions or small scripts to keys. In our project, it acts as the bridge between the voice/AI system and the game. Instead of directly automating the entire game, MacroKeybind triggers specific player-like actions, such as moving forward, turning, opening a menu, or switching items.

Challenges we ran into

One major risk was that the mod could cross the line from assistive technology into unfair automation. We addressed this scope issue by limiting the commands to actions a player could normally do themselves. The mod can help with basic controls, but it should not do things like find diamonds, search for hidden resources, or automatically pathfind across the world. We also included the idea of a clear “stop” command so the player can interrupt movement immediately, and we planned error handling for API failures, such as when a request is blocked or the AI service is unavailable.

What's next for Chat2Craft

With more time, we would build a stronger voice pipeline, add more customizable accessibility settings, and create a larger library of safe commands. We would also improve the command filter so the system can clearly distinguish between assistive requests and unfair automation. Additionally, we want to add support for as many versions of Minecraft as possible, and ensure compatibility with any additional mods to allow players to gain the entire Minecraft experience, regardless of version. Long term, we imagine this as a lightweight accessibility layer that players can simply install and launch.