The ARC AGI Knight Rises

Increasing Chat GPT Sol’s ARC 3 AGI score 6.5x to 50.37%, offering a framework for fluid intelligence, adaptability, and novel problem-solving with only a 100-line prompt.

Comment

Hackathon Summary

Many experiments and graciously accepted Tibo usage resets later, I, alongside my partner in crime Chat GPT Sol, created a zero-shot, lightweight harness captured in a <100 line prompt. This simple prompt, paired with Sol, led to a 6.5x increase in Sol’s performance on the ARC 3 AGI benchmark.

With a score of 50.37%, I'm not only proud that the score is competitive on the community leaderboards, but also that it was achieved with a general purpose solution not built specifically for this benchmark. Rather, I used ARC 3 AGI as a training ground to discover a promising framework to get agents to exhibit fluid intelligence, adaptability, and novel problem solving across domains.

Inspiration

I was inspired to try new processes with AI, apply research, and ultimately test the performance of my ideas:

  • For running fully AI automated experiments to discover & optimize solutions, Andrej Karparthy's 'autoresearch' experiment and Open AI's approach to discovering the proof for the Cycle Double Cover Conjecture
  • Based on my research with neuroscience, psychology, and biology, how the executive functions of the human brain (prediction, simulation, etc.) enable us to thrive in complex, uncertain environments
  • Finding ways to bridge the gap between human and AI agent performance on the ARC 3 AGI benchmark

What it does

The prompt gives the agent a light weight harness/loop that it must follow and is paired with a basic memory system. It took 150+ experiments over the last week to land on something as simple as ReAct but robust enough to handle the challenge of the ARC 3 AGI benchmark.

These are the steps it takes, documenting the output of each step into a log for reference:

  1. Observe
  2. Builds a model of the environment by identifying and documenting all objects (appearance, function, location, etc.)
  3. Orient
  4. Analyzes and plans around the current context to figure out what to do next: -- Plan: Select the next best action -- Predict: Create a hypothesis for what will happen based on the action -- Simulate: Construct a working model of the current state and play out what-if scenarios (algorithm, tool, image, document). Adjusts plan & and predictions accordingly before acting.
  5. Act
  6. Perform the action or action sequence
  7. Reflect
  8. Close the loop by determining: -- What changed in the environment -- Were predictions/simulations true or false -- What was learned

Some important details:

  • This structure is based off of the OODA decision framework (observe, orient, decide, act). The problem when applying it purely is humans have functions baked in to support this loop naturally whereas AI agents do not. I needed to determine the explicit behaviors that should be displayed at each step that made sense for the agent to display the right qualities.
  • This type of loop shouldn't be thrown at everything. Just like for humans, it requires excessive thought and effort for facing special circumstances. Otherwise, the default approach works great.

How we built it

I started by selecting one game from ARC for experimentation. The thought was if it could exhibit the right characteristics on one game, that structure could be applied to all (a sign of fluid intelligence at work). Once I was confident in the solution, I ran the full benchmark.

Sol built the entire benchmark testing harness and then helped with the following:

  • Used Sol Ultra to manage a portfolio of experiments to identify approaches with potential. The best run was 5/7 after 64 experiments.
  • Refined the best approach using Sol Max and a modified version of 'autoresearch'. Achieved the first 7/7 run after 20 experiments. Shifted to testing consistency with a fail fast approach.
  • When results plateaued, I synthesized the best ideas and hand wrote the next family of prompts. Achieved a 7/7 result on the first try.
  • Edited the prompt heavily to boil down to essentials without sacrificing performance, updated the agent orchestration approach. Used Sol to support with post testing evals and refinements.
  • Landed on a prompt that could achieve 7/7 runs several times in a row for a single game. I felt confident it was good enough for testing thoroughly and it achieved the results below.

Challenges we ran into

  • Achieving perfect and consistent runs on even 1 game of the benchmark: Eventually the performance of Sol generated solutions plateaued in performance. Instead of playing the odds and burning excessive tokens, I decided to shift Sol back to a partner instead of an automated experimentation manager. This is what unlocked the final solution.
  • Selecting the right agent design pattern & supporting memory structure: I tried single agent, supervisor, and sequential patterns. Multi agent conceptually is right to allow for specialization of functions, but the cost of speed/usage wasn't worth it. I settled on a single agent + basic memory log.
  • "Cheating": In the final scorecard, 2 subagents were on their last life and attempted to use the browser to desperately find a solution (against my rule). A handful of others got stuck (either with the TMUX interface or strategically) and reached out to the root agent for guidance. Is this cheating? For the official ARC AGI competition, yes. I let these actions slide because knowing when you hit an impasse and reasonably seeking help/resources isn't cheating, it is a display of intelligence.

Accomplishments that we're proud of

  1. Sol Max + my prompt scored 6.46x better than Sol Max + CoT on the ARC 3 AGI benchmark: My prompt: 50.37% link GTP 5.6 Sol Max + CoT: 7.8% link

What we learned

  • I continue to learn the line for where to fully automate to AI and when to step in - using AI as a partner instead. And sometimes, it's ok to not use AI at all and do things manually (although that feels wrong these days).

What's next for The ARC Knight Rises

  • Analyzing the traces of the to identify current gaps + next iteration of the prompt
  • Testing on tasks outside of the ARC 3 benchmark to understand real-world performance
  • Rerunning the ARC 3 benchmark with other levels of reasoning

Built With

Share this project:

Updates