BOB-Agent

178 tests passed; exp_001 remained champion.
exp_001 was accepted; exp_002 was rejected.
Five agents collaborate while the hidden test stays isolated.
All four safety and contract checks passed.

Inspiration

Recommendation research is not limited by the number of ideas. It is limited by how quickly a team can test an idea without losing reproducibility, leaking future data, or accidentally accepting a regression.

In a conventional workflow, the same person may change the model, adjust the data boundary, run the evaluation, and interpret the result. This enables fast iteration, but it also makes subtle mistakes difficult to detect.

We wanted to explore whether an agent-based development system could improve both experimentation speed and scientific discipline. BOB-Agent treats every experiment as a governed, reproducible, and reviewable state transition instead of an unstructured notebook or chat session.

What it does

BOB-Agent is a safety-gated five-agent system for autonomous recommender-system experimentation.

Each agent has an independent responsibility:

Agent A — Controller: plans experiments, approves scope, integrates evidence, and selects the champion.
Agent B — Harness and Reliability: validates contracts, runs approved experiments, and freezes reproducible artifacts.
Agent C — Data and Features: verifies temporal boundaries, leakage controls, and feature definitions.
Agent D — Models and Training: implements models, objectives, samplers, and training changes.
Agent E — Independent Evaluator: audits immutable predictions and independently verifies validation results.

For every experiment, Agent A freezes a hypothesis, baseline, single scientific variable, seed, training budget, and success rule. Agents C and D then review or implement the approved change. Agent B performs contract checks and a bounded synthetic smoke test before any real-data run is allowed.

Once the public-validation gate is satisfied, Agent B produces baseline and candidate artifacts from the same clean commit, dataset, features, seed, and budget. Agent E independently audits and evaluates the immutable predictions. Agent A then applies the predefined ACCEPT or REJECT rule.

Every decision is stored in experiment memory. A worse candidate cannot replace the verified champion, and hidden-test data remains outside the ordinary agent workflow.

How we built it

We built BOB-Agent with Python and NumPy for data processing, factorization machines, Bayesian Personalized Ranking training, and evaluation interfaces.

Codex CLI powers isolated and resumable agent execution. Each role works in a separate Git worktree and submits its evidence through a role-specific branch and pull request. Git and GitHub preserve provenance and make every handoff reviewable.

GitHub Actions run four required checks:

Unit tests
Repository-contract validation
Prediction-contract validation
Protected-file hash verification

JSON files define experiment specifications, configurations, state transitions, and handoff contracts. Append-only JSONL records preserve experiment history and manual interventions. Full Git commit SHAs and SHA-256 manifests bind every approval, configuration, prediction, checkpoint, and private artifact to exact bytes.

The coordinator identifies the next legal role, invokes it, monitors pull requests and CI, transfers private artifact references through verified manifests, and resumes interrupted campaigns. Unsafe changes, missing authorization, dirty worktrees, mismatched hashes, test access, or incomplete evidence cause the system to stop safely.

Our verified experiment trajectory currently includes:

Reference FM baseline: 0.601469 validation primary score.
exp_001 — ACCEPTED: replacing pointwise binary cross-entropy with same-user BPR reached 0.603871, an improvement of +0.002402. It became the new champion.
exp_002 — REJECTED: increasing BPR negatives per positive from one to two reached 0.603260, which was -0.000611 below the champion. BOB-Agent rejected the candidate and retained exp_001.

All model-selection results came from the public validation split. Hidden-test access remained disabled.

Challenges we ran into

The hardest part was not implementing the recommendation model. It was making agent autonomy auditable.

We had to prevent baseline and candidate routes from being exchanged, bind configurations to experiment identities, distinguish approval commits from artifact-producing commits, and verify artifact bytes before evaluation.

We also had to ensure that a synthetic smoke test could never be mistaken for formal model-quality evidence. Real validation required a separate recorded gate, while hidden-test evaluation remained unavailable to the agents.

Supporting asynchronous agents introduced additional challenges. Role sessions may be interrupted, Git branches may advance independently, and large predictions or checkpoints cannot be committed to a public repository. We addressed these problems with resumable sessions, isolated worktrees, fail-closed PR gates, and private SHA-256 artifact manifests.

Accomplishments that we're proud of

We reproduced the reference factorization-machine result within approximately 0.00013 primary score.

BOB-Agent then found and accepted a real BPR improvement, tested a plausible follow-up hypothesis, detected that it performed worse, and automatically retained the verified champion.

We are especially proud that failure became useful evidence instead of a lost experiment. The system preserves both successful and unsuccessful iterations, allowing future agents to learn from the complete research trajectory.

We also built:

A working five-agent experiment coordinator
Bounded multi-iteration campaign execution
Automatic role branching, pull requests, CI monitoring, and safe merging
Clean-commit and full-SHA provenance contracts
Temporal leakage and development-date controls
Immutable prediction and checkpoint auditing
Protected canonical evaluation files
A dataset-free demonstration workflow
Strict separation between public validation and hidden-test evaluation

The public repository contains no datasets, predictions, checkpoints, credentials, or private experiment artifacts.

What we learned

Agents become more useful when they are able to say “no” with evidence.

Explicit role boundaries and immutable handoffs made failures informative instead of ambiguous. Independent evaluation reduced the risk that the same agent would implement a change and then approve its own result.

We also learned that autonomy is not all-or-nothing. Bounded autonomy is more practical: the system can continue automatically when every requirement is satisfied, but stop truthfully when authorization, external data, or human action is required.

Most importantly, improving an autonomous research system is not only about increasing the model score. Reproducibility, honest rejection of regressions, traceable decisions, and protection of unseen test data are equally important.

What's next for BOB-Agent

Next, we plan to add managed notifications and secure cross-device artifact transfer so agents running on different machines can exchange large private outputs without placing them in Git.

We also plan to expand the research space with additional single-variable experiments, including time-safe historical preference features and alternative ranking objectives, while preserving the same independent evaluation and stopping rules.

Finally, we want to connect BOB-Agent’s existing label-free final-submission freeze to an organizer-controlled hidden-test evaluator. The system would select and freeze the final candidate by commit, configuration, checkpoint, and hash, while hidden labels and hidden-test metrics remain invisible to every agent.