Inspiration
Most research agents start from zero every session. Ask the same question twice and you get the same research process twice, even if the last answer turned out to be wrong. That felt like a real gap: agents that retrieve documents constantly, but rarely remember whether the sources they trusted last time actually held up.
The idea for Agent Black Box came from wanting to prove something narrower and more concrete than "the agent has memory." Lots of systems can store and display past outputs. Far fewer can show that retrieved memory causally changes what the agent decides to do next. That distinction, display versus decision, became the whole point of the project.
The demo domain, Crynux and the Neptune Cash and Neptune Privacy fork pair, was chosen because it offered something rare: a real, verifiable case of documentation drift. Crynux's older docs describe its nodes as Stable Diffusion image generation only. Current docs describe much broader LLM and VLM inference under a named consensus protocol. That is a genuine, checkable contradiction in the wild, not a synthetic test case.
What it does
Agent Black Box is a research agent that records every research task as a structured episode in CockroachDB, including which sources it used, what it concluded, and what later turned out to be wrong. Before starting a new task on a project it has researched before, it retrieves that history and lets it change its behavior: which sources it trusts, how it prioritizes them, and what it double checks.
Concretely, the loop is retrieve, plan, act, evaluate, learn, persist, run as a plain readable script rather than hidden inside a managed agent framework. When new research contradicts an existing claim, a lightweight contradiction check (vector search for the closest prior claim, then a single LLM judgment call only for genuinely close candidates) fires, dings the old source's reliability score using a deterministic exponential moving average, and writes a lesson. The next time the agent plans research on that project, it reads that lesson and visibly deprioritizes the source that produced the outdated claim, citing the specific reliability score and source ID in its own reasoning.
This was proven live across three real sessions against Crynux: Session 1 established a claim from a real source. Session 2, on a reworded query, retrieved that claim, found a genuine contradiction (not a trivial rewording, a real narrowing of scope), and dinged the source. Session 3's plan stage explicitly named the dinged source by ID and reliability score and deprioritized it. That is the entire thesis, demonstrated end to end on real infrastructure, not simulated.
A Next.js web UI provides two views: Ask, for submitting a research query and seeing the strategy, answer, claims, and lessons, and Memory Trace, which shows source reliability scores, recorded contradictions with old versus new claim text, and recent lessons, the single view that makes "the agent remembers and corrects itself" visible at a glance.
How i built it
The stack is intentionally small. AWS Lambda runs the agent loop as a stateless orchestrator, calling Amazon Bedrock (Claude Sonnet 5 via the Converse API, Titan embeddings) for reasoning and CockroachDB Cloud for memory. Two separate credentials enforce a real security boundary: the app backend writes through a read write psycopg connection only at persist time, while the agent's own runtime reads during planning go through CockroachDB's Managed MCP Server using a read only credential. Even a confused prompt or a bug in reasoning cannot write, because that credential is never given the ability to.
Reliability scoring is a simple exponential moving average, not a trained model, on purpose. It is explainable in one sentence and verifiable by hand, which matters more than sophistication for a system whose entire claim is that its behavior should be auditable.
Contradiction detection is deliberately narrow: vector search for the single closest prior claim, and only if the distance is under a threshold does it cost an LLM call to judge whether the two claims actually conflict. This kept the system fast and cheap rather than running full fact verification over every claim pair.
Deployment followed a two step verification approach: everything was built and dry run tested locally first with mocks, then verified against real infrastructure (a live cluster, a real Bedrock account, a real Lambda deployment) before being trusted.
Challenges i ran into
Nearly every integration boundary produced a real, specific bug, and the project's build log is honestly as much a record of those as of the architecture itself.
On the CockroachDB MCP side: the exact tool name (select_query, not a generic run_sql) and argument shape (query, database, and cluster_id as tool arguments, not sql and not HTTP headers) were both guessed wrong initially and only confirmed by fetching the real MCP docs and the live tools/list response rather than assuming. The MCP Streamable HTTP transport also requires a full initialize handshake with a captured Mcp-Session-Id, which the first version skipped entirely.
On the Bedrock side, embedding vectors formatted with Python's default float precision exceeded the MCP tool's 16384 character query limit; formatting to 6 decimal places fixed it with no meaningful loss of precision for nearest neighbor ranking. The Converse API's response also does not reliably put the model's text in content[0], which caused a string of confusing empty-string JSON parsing errors until every call site was hardened to collect all text blocks and fail loudly with real diagnostics instead of silently returning nothing.
Lambda deployment surfaced its own chain of issues: a Python 3.11 versus 3.12 compiled extension mismatch for psycopg, AWS_REGION being a reserved environment variable that cannot be set manually, Secrets Manager secrets stored as JSON objects rather than plain strings, a stray whitespace character in a copy pasted token, and two full rounds of TLS trust store debugging before landing on the certifi package's bundled CA file, since Lambda's minimal Amazon Linux runtime does not populate the OS certificate paths that worked automatically on Windows.
The Function URL itself needed both lambda:InvokeFunctionUrl and lambda:InvokeFunction resource policy permissions, a requirement AWS added in October 2025 that many older tutorials do not mention. And once the web UI was wired up, a native CORS configuration on the Function URL collided with the Lambda code's own manual CORS headers, producing a browser side "multiple values" error that curl never surfaced, since curl does not enforce CORS.
The single most valuable habit across all of this was refusing to guess twice. When an error recurred with identical text, the fix was always to pull the real CloudWatch traceback via aws logs tail rather than re-guessing at candidate causes, which is exactly how a stale, unfixed extractor.py file was finally caught after several rounds of chasing what looked like new bugs but was actually one old one that had never been redeployed.
Accomplishments that I'm proud of
Getting the core thesis, retrieved memory causally changing agent behavior, proven live on real infrastructure rather than simulated or mocked. It would have been easy to fake this with a scripted demo. Instead, three real sessions against a real, live Crynux corpus produced a genuine contradiction, a genuine reliability ding, and a plan stage that named the exact source and score it was reacting to, in its own words, unscripted.
Also proud of the security boundary actually holding up under implementation, not just existing on paper: the agent's own reads go through a credential that is structurally incapable of writing, which is a real, verifiable guarantee rather than a policy the code merely promises to follow.
What i learned
That most of the real difficulty in building an agentic system with genuine memory is not in the agent logic itself, it is in the seams between managed services: exact tool names, exact argument shapes, exact response formats, exact certificate paths. None of that is glamorous, and all of it will break a demo if left unverified. Verifying against the real service, every time something felt uncertain, rather than trusting a plausible guess, was the difference between a system that worked once in testing and one that actually held up under a live deployment and a live UI.
What's next for agent-black-box
Tightening the Function URL's authentication before treating it as a lasting public endpoint rather than a demo convenience. Extending the demo corpus to more projects to see whether the contradiction detection and reliability scoring generalize past the Crynux case they were built and proven against. And exploring whether the same two credential security pattern, a structurally read only path for the agent's own reasoning versus a separate write path gated to persistence, is worth generalizing into a reusable pattern for other agentic memory systems built on CockroachDB.
Log in or sign up for Devpost to join the conversation.