Unwritten

Unwritten learns the rules nobody wrote down—turning past team decisions into an evidence-backed Rulebook that helps agents act with organizational context.

Comment

Inspiration

Every team has rules nobody wrote down.

Humans learn them by working there: what gets accepted, rejected, escalated, treated as a duplicate, or considered outside the team's scope. Agents usually know how a tool works, but not how a particular organization actually uses it.

Unwritten asks a different question: instead of giving agents more static instructions, can they learn an organization's operating rules from its past decisions — and can we prove whether that memory actually helps?

What it does

Unwritten turns historical team decisions into a compact, evidence-backed Rulebook that an agent can use when making new decisions.

It learns across a three-round memory lifecycle:

V0 → V1 → V2 → V3

The Rulebook is deliberately bounded rather than becoming an unlimited transcript. Rules accumulate evidence, gain or lose confidence, merge, contradict one another, and can be retired.

The real training run uses:

  • 206 historical TRAIN examples
  • 12 learned active rules
  • a hard budget of 15 active rules
  • 66 DEV examples
  • 88 untouched FINAL_HOLDOUT examples

Unwritten also learns a tool policy: when its current evidence is sufficient, and when historical precedent should be retrieved.

GitHub issues are the benchmark adapter, not the product boundary. The same architecture can extend to support operations, approvals, incidents, sales operations, and other workflows where historical decisions encode institutional knowledge.

How we built it

Unwritten was built end-to-end with Agent Orchestrator.

I split the system into isolated AO worker sessions responsible for:

  • benchmark and evaluation foundation
  • real data materialization
  • Rulebook and learning lifecycle
  • historical-search tool policy
  • instrumentation
  • integration
  • real-model DEV evaluation
  • bounded debugging and improvement

Each worker operated in its own Git worktree, validated its work, produced a tested commit, and stopped with a structured handoff before integration.

That process mirrors the philosophy of Unwritten itself: isolate changes, preserve evidence, measure results, and never silently rewrite failed experiments.

Architecture

Historical decisions
↓
Frozen chronological TRAIN
↓
Feature abstraction
↓
Reflector / Rule induction
↓
Bounded Rulebook — V0 → V3
↓
Policy Agent
↓
Tool Heuristics
↓
Historical Search when useful
↓
Decision
↓
Evaluator + Instrumentation
↓
Failure analysis / bounded improvement

Historical retrieval is strictly temporal:

record.createdAt < currentIssue.createdAt

During DEV evaluation, only TRAIN history is available. DEV outcomes never update memory or become precedent for later DEV cases.

Evaluation

I designed the benchmark as a controlled bakeoff:

  • A0 — Cold: no history, no Rulebook
  • A1 — Raw-RAG: historical retrieval, no learned Rulebook
  • A2 — Scrambled memory: deterministic memory control
  • A3 — Unwritten: learned Rulebook + tool policy

The benchmark predicts historical resolution disposition for externally reported issues in the public modelcontextprotocol/inspector repository:

  • RESOLVED_COMPLETED
  • RESOLVED_NO_NEW_WORK

Real inference used Claude Sonnet 5 through isolated Claude Code CLI processes.

Every prediction was isolated. During the verified A3 runs:

  • web searches: 0
  • web fetches: 0
  • subagents: 0
  • DEV memory mutations: 0
  • FINAL_HOLDOUT accesses: 0

What the evaluation found

The most useful result was not a flattering benchmark number — it was a measurable failure that the system allowed us to diagnose correctly.

In the first complete A3 evaluation, Unwritten produced 66/66 valid DEV predictions, but its learned tool policy chose historical search on:

0 / 66 cases

The root cause was clear: the learned V3 rules were broad marginal rules whose confidence largely reflected the dataset's majority-class base rate. The tool gate therefore treated weak evidence as sufficient evidence.

I performed one bounded correction to the search decision logic without changing the model, benchmark, Rulebook contents, TRAIN data, prompt, retrieval top-k, ranking, or temporal filters.

After the fix:

Historical search: 0/66 → 66/66

The rerun completed 66/66 predictions with:

  • Macro-F1: 0.5307
  • Accuracy: 0.8333
  • Historical-search calls: 66/66
  • Web/search/fetch/subagents: 0

Macro-F1 remained unchanged.

That result was important: it falsified the hypothesis that missing retrieval was the primary bottleneck.

The next bottleneck is richer rule induction, not retrieval availability.

Rather than tuning repeatedly against DEV or touching FINAL_HOLDOUT, I stopped the experiment and preserved the negative result.

Challenges

The hardest part was preventing Unwritten from becoming a convincing-looking but scientifically weak demo.

The original benchmark target was rejected because issue templates made generic classification too easy for a cold model. I redesigned it around repository-specific operating decisions.

The system also had to defend against:

  • temporal leakage
  • cross-split duplicate families
  • future historical records
  • DEV outcomes becoming memory
  • model-session carryover
  • fabricated usage metrics
  • partial model-route failures

A real Claude CLI quota failure interrupted the first full bakeoff. Instead of pretending the run completed, it was classified as BLOCKED and preserved as evidence.

Accomplishments

  • Built from scratch during the hackathon using Agent Orchestrator
  • Frozen chronological benchmark from real public data
  • 206 TRAIN / 66 DEV / 88 untouched FINAL_HOLDOUT
  • Full V0 → V3 memory lifecycle
  • 12 active learned rules under a 15-rule maximum
  • Strict temporal historical retrieval
  • Real Claude Sonnet 5 evaluation
  • 163 passing tests in the final bounded iteration
  • Detected and corrected a real tool-policy failure
  • Restored retrieval coverage from 0/66 → 66/66
  • Preserved negative results instead of tuning them away
  • Kept FINAL_HOLDOUT untouched

What we learned

More memory and more retrieval do not automatically make an agent better.

Unwritten showed that a retrieval system can work perfectly mechanically while the real bottleneck remains the quality of the learned abstractions.

The next iteration therefore needs richer rule induction: multi-feature interactions, semantic abstractions, counterexample mining, and rules that represent organizational scope rather than surface-level correlations.

The broader lesson is simple:

An agent improvement system should be able to say, with evidence, “that change worked mechanically — but it did not improve the outcome.”

What's next

  • richer semantic and multi-feature rule discovery
  • counterexample-driven reflection
  • information-gain-based retrieval policies
  • selective abstention and human escalation
  • adapters for support, approvals, incident response, and operations
  • a complete single-route A0/A1/A2/A3 evaluation
  • one final untouched holdout evaluation

The long-term goal is not an issue classifier.

It is an organizational learning layer for agents.

Built With

Share this project:

Updates

Submission history