Agentic Chain-of-Ideas

Knowledge Graph Generation
Paper Emphasis
Hypothesis and Experiment Outline
Code Generation and Results
Experiment Insights

Inspiration

In recent years, Language Models have achieved breakthrough results on benchmarks. Reasoning models have demonstrated a large increase in performance - jumping from 13.4% (GPT-4o) to 56.7% (o1-preview) to 83.3% (o1) on the AIME Competition Math benchmark 0. They have also exhibited emergent behaviour which arise unexpectedly as models are scaled up in size and trained on more extensive datasets. These abilities are not present in smaller models 1.

But, PhD-level intelligence is not PhD-level utility. Companies hire PhDs to do research, not to answer examination questions. Thus, the next frontier for AI, and AGI in fact, is agentic research. We hypothesise the knowledge discovery process and develop a framework for LLMs to pursue autonomous science, which could exponentially scale the performance of research-first organisations and make every researcher a 10x researcher.

What it does

We hypothesise that Science is built upon interpolating ideas - chains of ideas. These manifest in 'hedgehog' and 'foxy' paradigms - interpolating between fields and within fields 2. By mining the most relevant research papers from Semantic Scholar, we generate a Knowledge Graph pertaining to a research field that the user is interested in. This graph is built by looking at the citations for a landmark paper related to the field, and their citations. How deep to build the graph is a tuneable parameter. The user can select papers to emphasise on, which are more relevant than those given by Semantic Scholar. This is then passed as context, together with the research papers, to Claude 3.5-Sonnet. Although we use Claude due to resource constraints, we make it easy to swap language models via our Python backend. Our goal is to develop an initial array of hypotheses and experiment plans.

The user will select an appropriate hypothesis and experiment plan. Early examination of language models, when used zero-shot without context, showed a tendency to hallucinate. Thus, when combined with Retrieval Augmented Generation with research reports and relationship modelling in Knowledge Graphs, we strengthen the ability of the LLM to generate a hypothesis with good grounding. This is accentuated by the increasing computational power of reasoning models. Given that Claude 3.5-Sonnet is not a reasoning model a la OpenAI-o3 or Deepseek-R1, we prompt Claude 3.5-Sonnet to use Chain-of-Thought to sequentially step through this generation process.

Afterwards, the experiment plan is approved and converted into Python code. We have created an inline code editor for the user to edit and approve the code. This is then run on the user's local compute and the results are displayed. They are then collated into a research report in Markdown.

How we built it

We used Python, Streamlit and Claude 3.5 Sonnet.

Challenges we ran into

We initially tried to use Claude Computer Use, but it was not great developer experience wise. It's in Beta anyway. We innovated and came up with another approach to run experiments that worked.

Accomplishments that we're proud of

It works for our dissertation topics :). Deterministic Concurrency, Continual Learning, Hardware Security, Underwater Computer Vision, etc.

What we learned

AGI may not be too far. And don't commit your keys to Github :)

What's next for Agentic Chain-of-Ideas

We propose future extensions - (1) to scale the knowledge graph to N graphs, enabling interpolation at scale. Presently we support only 1 knowledge graph, and this constrains the amount of interpolation and thus new ideas the LLM can seek. (2) We also aim to introduce a Markov reward strategy. We view hypothesis selection as a Markov Decision Process, and thus can utilise a Multi-Armed Bandit reward maximisation strategy to refine and select the best hypothesis in a continuous process. This will help enable continuous research and development at scale.