Minecraft Companion

Inspiration

As Minecraft player, if you lack available friends, you have less fun. Existing companion tools lack personality and in-game technical capability.

What it does

Minecraft Companion joins your server as a real player you talk to in chat or out loud over push-to-talk. You give it a name, backstory, personality, skin, and voice, and it follows you, fights mobs, mines, builds structures from natural language, and replies in the voice you picked. It has a live mood that drifts with what happens to it, mirrored on a physical Arduino LED on our desk.

How we built it

We built voice in input as a local STT server triggered by push-to-talk; voice out is ElevenLabs streaming TTS for replies. The LLM then utilizes a tool-calling loop that invokes actions via functions. The bot itself is a mineflayer body with pathfinder, pvp, collectblock, tool, armor, and auto-eat plugins. Character creation is a CLI. Mood is displayed by a mapping gameplay events to LED color, and gesture-based commands use an ultrasonic sensor on the same Arduino.

Challenges we ran into

Shaping LLM input to return scripts that created schematics capable of generating aesthetic Minecraft structures in minimal time.

Accomplishments that we're proud of

We built something that feels alive, where the combination of personality, voice, and the physical mood LED provides a unique and fulfilling experience. Another accomplishment is the effective schematic generation, diff planning, and parallel crew execution of Minecraft builds that we made work on arbitrary prompts.

What we learned

We learned to integrate an effective CLI into a larger app that benefited from it.

What's next for Minecraft Companion

Computer vision for smoother navigation, saving state between sessions, a marketplace where users could share their companions, and local model support.