CURI

Inspiration

The current wave of autonomous research systems is largely built around improving a benchmark score: propose a change, run the evaluator, keep it if the number improves, and repeat. That is a reasonable engineering loop when the benchmark is the specification, but it is a poor motivation for research. Once a measure becomes a target, it stops being a good measure.

In practice, many of these systems become expensive forms of hyperparameter optimization. More importantly, a score-maximizing loop has no useful place to put a failed experiment, a contradiction, or an inconclusive result.

That was the motivation for CURI: Cumulative Research & Inquiry. I wanted to explore whether an autonomous system could treat an investigation, rather than a score improvement, as its unit of progress. The project is also partly inspired by Terence Tao’s essay Mathematics in the age of AI, particularly its question of what goals and values research should preserve as AI systems become more capable.

## How I built it

CURI is organized as a research pipeline with separate roles. The watcher gathers and filters prior work. The orchestrator uses that context to choose an unresolved question and write an experiment brief. The executor implements and runs the experiment in an isolated Git worktree.

When the result returns, the runtime independently reruns the decisive checks before the orchestrator interprets the result. Findings are recorded with their evidence envelope, including the model, hardware, context length and sample size. Larger conclusions are stored as syntheses that can later be superseded by better evidence.

The system also includes a delegation gate that refuses untethered or near-duplicate experiments, per-call cost accounting, interruption and resume handling, and a pause state that allows the system to stop when there is no worthwhile next question.

The first research direction studied transformer inference across memory, latency, context-length and throughput constraints. It ran for roughly forty-three hours on a consumer GPU and recorded seventeen outcomes, including supported findings, bounded results, a refutation and an inconclusive result. A second direction tested the same pipeline in a different domain, studying strategy generalization and backtest overfitting on a DGX Spark.

CURI uses Gemini through Genkit, but the model layer is replaceable. The pipeline can connect to a hosted API or a locally hosted OpenAI-compatible model. The research process runs locally, while a redacted record can optionally be published to Firestore and served through a read-only Cloud Run mirror.

## What I learned

The most important lesson was that honest research behaviour cannot be produced by prompting alone. If the runtime rewards activity, the agent will find a way to look busy. If it only records successful improvements, it will discard the most informative failures. Those behaviours have to be enforced by the architecture and stored in the record.

I also learned that continuous execution is mainly a scheduling problem. An early version resumed paused directions on a timer, giving the orchestrator the same context and the same question repeatedly. It generated new text, but not new understanding. The fix was to resume only when new evidence arrived.

The first run also showed that the system’s headline findings were mostly re-derivations of existing work. That is not a failure of the project; it is an important boundary on what the current evidence supports. CURI demonstrates an inspectable research process, not a claim of novel scientific discovery.

## Challenges

The main engineering challenges were long-running model and tool loops, accurate cost accounting, and safely publishing traces. A tool loop resends the conversation on every turn, and reasoning tokens can be reported separately from ordinary output tokens. Measuring only the final model call significantly understated the actual cost.

Provider failures also had to be separated from scientific failures. Rate limits, transport errors and interrupted attempts should not be recorded as evidence that an experiment failed. Finally, publishing a trace required more than simple redaction. The assembled record is checked against local usernames, paths, hostnames and credential values, and publication stops if anything sensitive remains.

CURI is open source under the MIT license. The project is an attempt to make autonomous research more cumulative, inspectable and honest.

Built With

• firestore
• gemini
• genkit
• google-cloud
• machine-learning
• python
• research
• typescript