Llama Voyager

Voyager, the breakthrough paper on open-ended embodied agents with LLMs now runs on Llama-3 and has comparable results to GPT-4 on the skill curve thanks to Groq's inference speed.

Comment

Inspiration

Inspired by Voyager and wanting to test the limits of Llama3, we decided to port this project this new model.

What it does

An embodied agent joins Minecraft, an open world sandbox game, and starts learning skills of navigating this world with chain-of-thought inferences and code generation. Once a code snippet is succesful in performing a desired action in game, it is saved as a skill for the bot.

As the bot learns more skills, it can take actions significantly faster in game.

How we built it

We started with the Voyager repo on Github and made adjustments required to run Llama3 running on Groq.

Challenges we ran into

Out of the box, performance on Llama3 70B was very bad. It would take a long time before the agent was able to develop a skill due to poor output from Llama and high iteration steps. We started prompt engineering the skill generation / code gen inference for the agent and added Groq to increase inference iteration speed. After this, the agent started behaving well and actually developing skills in a human comprehensible amount of time :D

Also the whole set up of having the agent actually show up and do things inside Minecraft controlled by the Llama Voyager server was very challenging, took most of our day 1!

Accomplishments that we're proud of

Actually having a functioning demo of an agent learning skills of navigating Minecraft powered by Llama 3 running on Groq!

What we learned

It's going to be a big effort to run this on a smaller model but we are going to try!

What's next for Llama Voyager

Finetuning 8B to match this performance!

Built With

  • groq
  • llama-3
  • minecraft
Share this project:

Updates