TraceGuard

Inspiration

Supply-chain recalls are a race against two clocks at once: the clock on getting a defective batch off trucks before it reaches customers, and the clock on every other warehouse in the network that doesn't yet know the batch is bad. Most systems handle this with application-level locks, message queues, or "eventually consistent" reconciliation jobs — which means there's always a window where a warehouse can ship a batch that was just quarantined. We wanted to prove that window doesn't have to exist if the guarantee lives in the database's own transaction semantics instead of in application code.

What it does

TraceGuard is an autonomous agent that decides when a batch should be quarantined and executes that quarantine atomically — blocking every in-transit shipment of that batch in one transaction, while guaranteeing no warehouse can create a new shipment of that batch after quarantine commits, even under a genuine concurrent race. The agent reasons over live and historical defect reports using semantic search over CockroachDB's vector index, recalls similar past incidents through CockroachDB's Managed MCP Server, and triggers the actual quarantine through an independently deployed AWS Lambda function. A React chat UI sits on top, showing an operator exactly which backend system handled each step of the agent's reasoning — CockroachDB SQL, CockroachDB MCP vector recall, or AWS Lambda — plus a live "race" trigger that spins up concurrent warehouse workers against a real quarantine to prove the guarantee on demand.

How we built it

Backend: FastAPI wrapping a tool-use agent loop (Groq-hosted or local Ollama), CockroachDB Serverless as the single source of truth, psycopg2 for direct writes and SELECT ... FOR UPDATE locking, and CockroachDB's Managed MCP Server for the agent's semantic-recall tool calls. Embeddings run locally via FastEmbed, so no data leaves the process for that step. The quarantine transaction itself lives in its own AWS Lambda function, invoked over boto3, kept deliberately separate from the agent's reasoning loop so the ACID-critical code path is independently deployable and testable. Frontend: React, TypeScript, Vite, Tailwind. Deployment: Docker image built with buildx and pushed to Docker Hub, served from Render; frontend on Vercel.

We validated the core guarantee two ways: a race_demo.py harness that releases N warehouse workers and one quarantine call from a shared barrier simultaneously across repeated trials, and a chaos_demo.py that SIGKILLs the quarantine mid-transaction and verifies the system recovers to a consistent state on retry — no stuck or ambiguous shipments.

Challenges we ran into

Nearly every hard bug lived at an integration seam, not in our own logic. Groq's tool-call parser rejected the model's own generated arguments when it emitted a numeric parameter as a string, failing before our code ever saw the request. The CockroachDB MCP server returned a generic unauthorized on EXPLAIN queries that turned out to be a Cloud RBAC role gap — the service account needed Cluster Admin or Cluster Operator, not just a SELECT grant, which we chased down the wrong path first. Once auth was fixed, a relation does not exist error surfaced a second bug: our tool-selection logic matched on the substring "query" and grabbed explain_query instead of select_query because of list order, and neither call was passing the database parameter the MCP tool schema actually required. Separately, the LLM itself occasionally hallucinated a placeholder string instead of chaining the real UUID returned by a prior tool call — a model-behavior problem we mitigated with input validation and tighter tool descriptions rather than trying to prompt our way out of it entirely.

Accomplishments that we're proud of

The safety guarantee has zero application-level locking code — it's a direct, provable consequence of both create_shipment and quarantine_batch opening on SELECT ... FOR UPDATE against the same row. We didn't just claim that in a doc; race_demo.py and chaos_demo.py actually exercise it under real concurrency and real process kills. We're also proud of keeping the entire AI reasoning path on local embeddings plus either local or generously-free-tier hosted inference — no Bedrock, no paid enterprise AI service anywhere in the stack — while still shipping a full agent + API + UI, not just a backend script.

What we learned

Generic error messages from managed services (a bare "unauthorized" or "Server returned an error response") are rarely the real story — the fix was almost always one or two layers deeper than the surfaced text suggested, and swallowing exceptions inside a tool-dispatch loop (to keep the agent's tool loop resilient) meant we had to deliberately add logging back in to see what was actually failing. We also learned to distrust an LLM's tool-call argument chaining even when the schema is well-typed; validating at the boundary (UUID format, int coercion) caught failures the model itself wouldn't have surfaced clearly.

What's next for TraceGuard

Tighten the CORS policy from wildcard to the deployed frontend's exact origin now that both are stable. Expand historical defect seed data beyond the current small sample set to make semantic recall meaningfully useful rather than illustrative. Add authentication to the API before any real multi-tenant use. Extend the quarantine cascade to notify downstream systems (e.g., a webhook or SNS topic) once a batch is quarantined, and explore whether the same row-lock pattern generalizes to other supply-chain invariants beyond quarantine-vs-shipment.

Built With

• amazon-web-services
• cockroach
• docker
• groq
• ollama
• python
• react
• render
• tailwind
• vercel