Mr. J

Introduction

AI has become increasingly more prevalent and many have wondered how this new technology will affect automation and our everyday lives. Mr. J demonstrates how this technology can be used to automate repetitive tasks in Minecraft so that the player can focus on the more creative aspects of the game. In addition to improving the user experience in one of the most well known video games, this project demonstrates the benefits of AI automation in the real world. Projects like Mr. J pave the path for future AI that can automate repetitive tasks in our lives so that we can focus on more difficult cognitive tasks.

What it does

This is a minecraft bot that helps alleviate some of the menial tasks within the game with LLM integration that will talk to the user in natural English while also providing a back-up to incorrect task declaration. If the user inputs a task incorrectly or tries to run a task that doesn’t exist, our LLM will trigger a task from our list that does work.

A list of tasks is as follows: Move: Moves to location; bot move x y z Hello: Says a message from our LLM integration; bot hello SetHome: Set’s the coordinates of the bot in question as the home for bots to return to; bot sethome GoHome: Causes bots to return to the previously saved home location; bot gohome Say: Causes the bot to say a message; bot say message Come: Causes the bot to come to the users location; bot come Drop: Causes the bot to drop a block of the specified block type; bot drop itemType Remember: Causes the bot to remember whatever is specified to be recalled later; bot remember string of information ValueOf: Recalls previously saved information; bot valueof Follow: Causes bots to follow the specified user; bot follow user Attack: Causes bots to attack the specified user; bot attack user GatherFood: Causes bots to attack a specific type of food; bot gatherfood pig Inventory: Causes bots to list their inventory; bot inventory Status: Causes bots to list their current status; bot status Equip: Causes bots to equip a pickaxe if in their inventory; bot equip Sword: Causes bots to equip a sword if in their inventory; bot sword Mine: Causes bots to mine one block; bot mine ChestItem: Causes bots to go to the nearest chest and receive items; bot chestitem itemtype number

The minecraft bots can work in plural and tandem using the swarm command. You treat the swarm command the same way you treat bot. Simply input swarm command and they will all act out the command together.

How we built it

We began the project by programming the bot to enter our Minecraft server and creating the basic functionalities like moving and jumping. After this, we used Mineflayer.pathfinder to help the bot find the most optimal path to get to a location. We created functions that told the robot to go to a specific location, go to a preset Home location, and follow players around.

After this we tested functions that interacted with the world such as mine and gatherFood. Both of these functions had to locate a target such as a block or animal so that the bot could perform its action. After a block was mined or an animal was killed, the bot automatically searches the area for dropped items that it can pick up. These functions interacted with other functions like attack and with the inventory to select the correct tool.

We built the LLM through trial and error. First, trying different ways of hosting the model on our computers, locally, through a flask API, or on supercomputer clusters. None of which worked, finally settling on using a model’s API offloading the processing to their server and using allotted tokens. This comes with the downside of token prices and needing the application to be online for it to function, but frees up a dramatic amount of processing power on our machines. The next difficult part was engineering the prompt to give us the information we wanted. We iterated drafts of the prompt throughout the time we were allotted slowly fine tuning the results to better fit what we needed. The output was required to be highly specific for our code to run, and to keep the natural language side of the LLM.

Challenges we ran into

The strict time deadline and number of functions the bot required caused pressure on us. We needed to develop a lot of simple actions and then quickly implement them into the more complex functions, where the complex functions would always reveal more bugs that were hiding in the simple implementation.

The LLM integration proved to be far more challenging than any of us anticipated. First, we tried to host a model on our system, but the model we chose at first was simply not powerful enough to be useful. Trying larger models proved to slow down our machines too much to be remotely usable. To combat this, we tried hosting them on a local api, but this did not give us any speed back. Finally, we realized we had to use an API and take the always online route. After finally getting the LLM to work and communicate with us, we had to engineer the prompt. This proved to be rather time consuming as even when the LLM produced correct results, the eval function would simply not evaluate it correctly. There were hidden characters or various data types that Javascript’s eval function refused to evaluate correctly. Eventually we managed to get the LLM to provide the correct answer in the correct format most of the time, however, with such a short time we were unable to get the model and prompt at 100% efficiency.

About halfway through the allotted time, we were faced with a difficult decision. As our LLM plan was failing and we were left with a lot of broken code, we needed to decide what path to take. Deciding that the LLM as it stood might not be the path, we diverted attention elsewhere because coming back to it. Changing plans and refactoring our team goal was a constant difficulty for us that we overcame to create our final product that we’re proud of.

Accomplishments that we're proud of

The amount of code we were able to write in such a short amount of time. Even under stress we were able to work together and create a large amount of functionality for our bots.

A well engineered LLM prompt that gives correct format in its replies. One of the more finicky and difficult parts of this project was dealing with and factoring the LLM, getting it to work and having it evaluate as functions was the most satisfying moment of the entire Hack-a-thon for me personally.

Having multiple bots work in tandem using swarm to accomplish many simple tasks takes our bots and deepens their complexity.

What we learned

Every project will inadvertently run into far more unexpected events that your team will have to find solutions for on the fly. No plan ever stays together and works right off the bat. Flexibility and on the fly thinking is an all important skill to have in a group setting.

Before starting this project, almost all of us had very little to no experience in Javascript or AI programming. We’ve all learned a large amount about dealing with asynchronous functions, prompt engineering, AI bots, LLMs, using multiple libraries in tandem to make a better whole,and the javascript language.

The strict time limit forced us to work together as a team and learn better communication. We were forced to face stress, failure, and things falling apart numerous times before coming together and finding solutions.

What's next for Mr. J

We will continue to work on the functionality of the code, implement new functions for our bots in game, continue to tune the LLM prompt, and iron out more of the lingering bugs. Our next steps are to expand upon the functions that are already written and tune the LLM so that the bots are easier to interact with. We would also like to add more functions so that Mr. J can help with even more aspects of the game. Mr. J is able to automate the repetitive and monotonous aspects of Minecraft and we want to make the user experience as streamlined as possible.

Built With

• gemini-api
• javascript