description
architecture
starting decision
authorize refused
promote refused
missing evidence refusal
deployment query
integrity query
discharge receipts
rankk 573 falsifier
retraction refused
safe cascade
final audit
proof overview

Inspiration

Solvent began with an earlier project of mine: the Elephant Bridge Protocol (EBP) in PithomLabs/workbench.

EBP started from a problem I kept encountering in research: an idea can be interesting without being proven, and new evidence can weaken an idea without making the idea worthless. I wanted a system where ideas could enter freely, accumulate explicit evidence debt, and become eligible for promotion only after that debt had been addressed.

The core principle was:

Entry is cheap. Promotion is expensive.

An objection should not simply erase an idea. A counterexample should create debt. A promoted idea should remain reopenable when new evidence arrives. Promotion should mean that current operational obligations have been retired, not that the underlying idea has become metaphysically true.

That work was initially motivated by questions in physics. I was thinking about hypotheses, competing explanations, null models, formalization, and the possibility that research systems can accidentally turn an interesting candidate into an assumed truth simply because it survived long enough.

Then I asked a more practical question:

What happens when an AI agent uses a belief as the basis for an action?

That changed the problem.

An agent can retrieve a plausible document and still miss the decisive counterexample. It can ask the wrong question. It can reason incorrectly over valid evidence. It can remember yesterday's conclusion after today's evidence has changed.

The dangerous transition is not retrieval.

It is authority.

That became the central idea behind Solvent:

Retrieval can be wrong. Judgment can be wrong. Authority cannot silently outlive the belief it depends on.

Why the project asks “Is etcd v3.5.x safe to deploy?”

I did not choose that question arbitrarily.

I wanted a real-world decision with three properties:

there had to be a concrete claim that could become authoritative;
the underlying evidence had to be large enough for semantic retrieval to be genuinely imperfect;
there had to be documented historical evidence capable of reopening the claim.

I considered Kubernetes because it would have been the obvious "production infrastructure" story. I rejected it because Kubernetes is too broad for the experiment I wanted to isolate. There are controllers, workloads, APIs, operators, networking, scheduling and many other moving parts. A three-minute demo could easily become a Kubernetes demo rather than an experiment about memory and authority.

I settled on etcd because it gives me a much cleaner experimental instrument.

etcd is focused enough that I can ask one concrete question about deployment safety, yet its public history contains exactly the kind of evidence trail I need: releases, upgrade problems, recovery failures, security issues, and data-integrity incidents.

Most importantly, the motivating contradiction is real historical provenance, not a fictional failure invented for a hackathon.

During the research that led to Solvent, I found etcd issue #13766 and the public maintainer discussion around it. In March 2022, etcd maintainers announced that recent reproduction of data-inconsistency problems meant that v3.5.0–v3.5.2 were no longer recommended for production. They described a scenario in which, under high load, killing an etcd process could leave committed transactions absent from some cluster members. ([Google Groups][1])

The subsequent etcd postmortem explains the underlying failure: a v3.5.0 refactor around the consistency index could allow the database state and WAL state to become inconsistent, with different members potentially reaching different states. The postmortem also records that the issue was difficult to reproduce and that the v3.5 release had not been qualified as comprehensively as earlier releases.

The maintainers later reported that the issue was addressed in v3.5.3/v3.5.4. ([Google Groups][1])

That history gave me the seed for the Solvent experiment.

I did not take a historical advisory and pretend it was a current statement that every v3.5 release is unsafe. The demo intentionally creates a decision state around the broader proposition “upgrading production etcd to v3.5.x is safe” and then asks whether an agent can earn authority for that proposition from evidence.

That distinction matters.

The purpose is not to re-litigate whether a particular patched etcd release is safe today. The purpose is to test whether an agent can distinguish retrieval, evidence, judgment, and authority when a historically documented counterexample exists.

I then built a real corpus from the etcd issue history. The measured corpus contains 7,239 issues after excluding pull requests.

That gave me a provenance chain:

EBP research idea
      ↓
belief + debt + promotion
      ↓
question: how does this work for an AI agent?
      ↓
real historical etcd incident
      ↓
7,239-issue evidence corpus
      ↓
retrieval experiment
      ↓
database-enforced authority

That chain is important to me because I did not want Solvent itself to violate the principle it was trying to demonstrate.

The project's own ideas needed provenance.

Why CockroachDB is not just hackathon pandering

I also considered what would happen if I simply added CockroachDB to a conventional RAG system because the challenge required it.

That would have been easy.

I did not want to do that.

The question I asked was: what does the agent actually need from its memory layer that a normal vector database does not give me?

The answer is that Solvent needs one durable system to contain:

embeddings,
evidence,
beliefs,
obligations,
action state,
provenance,
and transactional invariants.

The database is not merely remembering what the agent saw.

It is enforcing what the agent is allowed to do because of what it believes.

That made CockroachDB architecturally meaningful rather than decorative.

What it does

Solvent is a persistent memory and authority layer for AI agents.

The concrete demo asks:

Is etcd v3.5.x safe to deploy?

The system begins with a belief carrying six review obligations.

Before those obligations are discharged, the database refuses promotion:

23514 · promoted_is_debt_free

Authorization is independently refused:

23503 · gate

A third attempt to discharge the contradiction sweep without an evidence citation is also refused.

These are not UI messages pretending to be safety. They are database outcomes surfaced by the application. The captured deployment state shows the authorization refusal as a foreign-key failure on action_intent.

Then I search the real etcd corpus using CockroachDB's native vector index.

The deployment question surfaces plausible upgrade material, including:

#19220 · 0.372424

A different question focused on data integrity surfaces:

#14139 · 0.199509

This is the pivotal observation.

The memory did not change. The question did.

The deployment query is semantically closer to release and upgrade material, while the integrity query surfaces the data-inconsistency incident much more strongly. My research explicitly measured this behavior rather than assuming it.

That means an agent that asks only:

“Is this safe to deploy?”

and trusts the top-k vector results can miss the evidence that matters most.

Solvent therefore makes retrieval a source of evidence, not a source of authority.

I record explicit evidence receipts, retire the six obligations, promote the belief, and authorize the deployment.

Then I introduce the crucial falsifier:

#13766

It was not surfaced by the original deployment query.

That is intentional.

Its measured offline rank in the captured experiment is 573 / 7,239.

Now the belief can be wrong.

Solvent does not pretend otherwise.

I try to retract the belief while the deployment intent is still live. The database refuses:

23514 · live_requires_promoted

I then cancel the intent first and retract the belief. That transaction commits, leaving no live authority resting on the retracted belief.

Finally, I run the control experiment:

Schema	Isolation	Observed
naive	READ COMMITTED	no error, both transactions commit, AUDIT = 1
naive	SERIALIZABLE	`40001 RETRY_SERIALIZABLE`
hardened	READ COMMITTED	`23503 · gate`

The first row is the reason Solvent exists.

Correct application logic is not enough if the persistence layer permits an invalid committed state.

How I built it

I built Solvent in Go with CockroachDB serving two roles simultaneously:

persistent agent memory and the final authority boundary.

The architecture is:

Agent
  ↓
MCP server
  ↓
Solvent kernel
  ↓
CockroachDB

The agent reasons and proposes.

The MCP layer translates tool calls.

The Go kernel owns transaction discipline and belief-graph traversal.

CockroachDB owns the final legality of the committed state.

Three database invariants drive the core:

`promoted_is_debt_free`

A belief cannot become promoted authority while review obligations remain unresolved.

`gate`

A composite foreign key makes a live action intent impossible unless the referenced belief exists in promoted state.

`live_requires_promoted`

A belief cannot be retracted while live authority still depends on it.

The recursive belief traversal itself is application logic. CockroachDB does not walk the belief graph; it enforces that the transaction cannot finish with a live intent resting on an invalidated belief.

For retrieval I use:

CockroachDB native vector indexing
VECTOR(1024)
Amazon Bedrock amazon.titan-embed-text-v2:0
7,239 real etcd issues
scenario-scoped retrieval
explicit citation receipts

For deployment I use AWS App Runner and ECR, with ccloud as a fail-closed deployment preflight.

The two CockroachDB tools I actually verified are Distributed Vector Indexing and ccloud CLI. I deliberately do not count the schema mechanisms as additional rubric tools because they are the architecture itself.

The MCP layer is deliberately thin. It does not decide that a belief is promotable or that an action is safe. It validates, invokes the kernel, and faithfully surfaces the result.

Formal model

I also formalized the core belief/authority state machine in Lean 4 + Mathlib under formal/lean/.

The formal model mirrors Solvent's core vocabulary — beliefs, debt, intents, statuses, and dependency edges — and machine-checks that successful state transitions preserve the three core authority invariants:

promoted_is_debt_free
gate
live_requires_promoted

It also proves the central authority properties, including:

live_intent_implies_promoted
no_live_intent_on_retracted_belief

and the layered promotion/cascade properties:

promotion_updates_dependent_intent
cascade_retraction_updates_dependent_intent
cascade_retraction_cannot_leave_live_intent
cascade_update_preserves_gate
cascade_retraction_blocks_live_intent

The promotion and cascade proofs explicitly model the status propagation corresponding to the database's composite foreign-key cascade.

All formal theorems compile with zero sorry, zero admit, and zero unchecked axiom. The Lean toolchain and Mathlib revision are pinned for reproducibility.

The formalization is deliberately scoped:

Lean proves the abstract state-machine preservation properties. CockroachDB proves the real engine's refusal behavior empirically.

This is not a machine-checked refinement proof of the Go/CockroachDB implementation.

Challenges I ran into

The hardest challenge was deciding where authority should actually live.

The easy implementation would be:

if belief.Promoted {
    authorize()
}

That breaks down as soon as concurrency, unexpected state, cascades, or another code path enters the picture.

I wanted the invariant to survive correct application code encountering an unexpected state.

That pushed the final boundary into CockroachDB.

The second challenge was retrieval.

My original instinct was to improve retrieval until the decisive evidence appeared near the top.

The research showed me that this would actually weaken the project.

The honest retrieval miss is a better demonstration.

The deployment question returns genuine deployment-risk material, but it does not surface the historical data-inconsistency incident. The integrity-specific query does. That is a measurable property of semantic retrieval, not a contrived failure.

The third challenge was preserving corpus provenance.

I did not want to say “7,239 issues” based on a live GitHub counter. I built a measured corpus, excluded pull requests, used stable creation-time pagination, and recorded the resulting snapshot with a digest so that the same artifact could drive local and cloud ingestion.

The fourth challenge was resisting the temptation to turn the experiment into a generic AI-memory platform.

I kept the kernel domain-agnostic and chose one extremely concrete scenario so the project could demonstrate the architecture rather than merely describe it.

Accomplishments that I am proud of

I am proud that Solvent demonstrates multiple independent database-enforced refusals rather than a single happy path.

I am proud that I kept the retrieval failure visible.

I could have hidden the fact that the deployment query misses the important counterexample.

Instead, I preserved the result because it is precisely what makes the authority problem real.

I am especially proud of the control experiment.

The same race produces:

naive + READ COMMITTED
→ no error
→ both transactions commit
→ AUDIT = 1

while the hardened schema produces:

23503 · gate

That makes the argument much stronger than simply saying “the schema has a constraint.”

I can demonstrate what happens when that constraint is removed.

I am also proud that the project now has two independent forms of verification.

The CockroachDB proof is empirical: the real engine produces the refusal SQLSTATEs under the unsafe ordering.

The Lean proof is mathematical: the abstract state machine is machine-checked to preserve the same authority invariants across its successful transitions.

Neither replaces the other. Together they make the boundary clearer: the formal model tells me the state machine's intended semantics are internally consistent, while the database experiment tells me the actual target engine enforces the critical refusal behavior I depend on.

I am also proud of the continuity from EBP to Solvent.

EBP gave me the idea that new evidence creates new debt rather than simply erasing an old idea. EBP explicitly treats objections, counterexamples, null models, and failed assumptions as productive sources of debt.

Solvent takes that philosophy one step further.

In EBP, a wrong or incomplete belief might waste research time.

In an agentic system, a wrong belief can authorize an action.

That is a much higher bar.

What I learned

I learned that memory and authority are different concepts even when they belong in the same database.

Memory answers:

What did I retrieve?

Authority answers:

What am I actually permitted to do now?

I learned that retrieval quality does not equal epistemic safety.

An agent can retrieve excellent evidence and make the wrong judgment.

An agent can ask a poor question and miss decisive evidence.

No embedding model removes that distinction.

I learned to be suspicious of silent success.

A failure announces itself.

A bad transaction that returns successfully can become part of reality before anyone notices.

That is why the AUDIT = 1 control result is so important to me.

Formalization sharpened that lesson further. Once I wrote the authority state machine in Lean, I had to state exactly what Solvent guarantees and what it does not. That forced the boundary to become precise rather than rhetorical.

Most importantly, I learned that provenance has to apply to the ideas behind the system itself.

Solvent's origin is not:

“I invented a clever agent-memory architecture.”

It is:

EBP
→ evidence debt / promotion
→ question about agent authority
→ historical etcd failure
→ empirical retrieval experiment
→ database authority boundary
→ formal state-machine model

The project itself follows the epistemic discipline it advocates.

The historical etcd incident is documented.

The corpus is measured.

The retrieval miss is preserved.

The falsifier is not hidden.

The control experiment is reported even though the naive schema fails.

The formal model is explicit about what it proves and what it does not.

That makes the story considerably stronger than importing a convenient example after the architecture was already designed.

Real-World Impact

I chose software deployment because the consequences are easy to understand, but the underlying problem is much broader.

The deeper pattern is:

knowledge
   ↓
judgment
   ↓
authority
   ↓
action

That pattern exists anywhere an AI system makes consequential recommendations.

Physics and scientific research

This is the closest extension because it is where EBP began for me.

A scientific agent could maintain hypotheses, supporting evidence, counterexamples, null models, formalization status, and unresolved objections.

A hypothesis could be promoted as a working research artifact without being treated as established truth.

New evidence could create new debt and automatically reopen the hypotheses and downstream conclusions that depended on it.

That is much closer to a genuine research memory than a vector database full of papers.

Law

A legal agent could maintain statutes, cases, contrary authorities, jurisdictional assumptions, dates, unresolved conflicts, and the claims derived from them.

A retrieved case would not automatically become a safe basis for a recommendation simply because it ranked highly.

Medicine

A clinical reasoning system could preserve the evidence, patient context, assumptions, contraindications, and unresolved uncertainty behind a recommendation.

New evidence could invalidate the premise while preventing a stale recommendation from silently remaining actionable.

Finance and compliance

An agent could track regulations, interpretations, approvals, exceptions, evidence, and downstream decisions while ensuring that an invalidated interpretation cannot silently continue authorizing work.

Engineering and operations

This is the immediate extension of the etcd story: infrastructure changes, security remediation, incident response, deployments, and operational approvals all contain decisions that can become stale while dependent actions remain live.

The common question is:

What happens when the knowledge underneath an authorized action changes?

That is the problem Solvent is designed to address.

What's next for Solvent — Persistent Memory for Agentic Authority

I want to turn Solvent from a hackathon demonstration into a reusable authority layer for agentic systems.

The next step is to make the evidence feeds interchangeable while preserving the same core semantics:

capture evidence
      ↓
form belief
      ↓
track obligations
      ↓
retire debt
      ↓
promote
      ↓
authorize
      ↓
reopen when evidence changes
      ↓
safely unwind dependent authority

I want to explore richer provenance, stronger multi-agent isolation, finer-grained authorization, and better representations of uncertainty.

I also want to test the model outside software engineering, particularly in scientific research and law, where the original EBP motivation naturally connects to Solvent.

The larger goal is not to build an agent that is magically never wrong.

It is to build an agent whose mistakes remain visible, whose evidence remains traceable, whose authority is bounded, and whose memory can change without leaving yesterday's permissions alive in today's world.

How this maps to the judging criteria

Agentic Memory Design: CockroachDB is not a passive store. It holds embeddings, evidence, beliefs, obligations, provenance, and transactional action state while enforcing the transition from belief to authority. The vector corpus contains 7,239 real etcd issues, rather than a toy dataset.

Technical Implementation: I use CockroachDB's native distributed vector index for live ANN retrieval and ccloud as a fail-closed deployment preflight. The MCP boundary is deliberately thin, and the database invariants remain below the agent rather than being delegated to prompting.

Real-World Impact: The immediate application is deployment safety, but the model generalizes to any workflow where AI converts knowledge into consequential authority: scientific research, law, medicine, compliance, finance, security, and operations.

Production Readiness: The design explicitly considers transactional authority, provenance, isolation, fail-closed deployment checks, reproducible corpus ingestion, audit trails, and what happens when a previously valid belief becomes invalid. The corpus ingestion path uses a byte-identified snapshot rather than silently rebuilding different datasets in different environments.

Creativity & Originality: The project is not simply “RAG with CockroachDB.” The central idea is to treat retrieval, belief, and authority as different states, then make the database enforce the boundary between them. The conceptual lineage from EBP makes that design deliberate rather than incidental.

[1]: https://groups.google.com/g/etcd-dev/c/sad9tgmKU7Y/m/-T-4sgTDAQAJ?utm_source=chatgpt.com "Etcd v3.5.[0-2] is not recommended for production"

Built With

amazon-web-services
bedrock
ccloud
cockroachdb
docker
ecr
go
mcp
taskfile

Solvent — Persistent Memory for Agentic Authority