Inspiration
Physical AI agents often behave as if every execution were their first. A robot may encounter a hazardous region, terminate, restart, and repeat the same mistake because the failure was stored only in transient logs rather than converted into actionable memory.
We built CoreSense to explore a simple question: Can an agent preserve a physical failure across process boundaries, retrieve it when a similar situation occurs, and use that memory to make a better decision?
CoreSense turns a structured execution outcome into durable semantic memory. Instead of merely storing telemetry, it preserves what happened, why it failed, what evidence supports the record, and what bounded recommendation should be considered during the next run.
What it does
CoreSense demonstrates a complete failure-to-improvement memory loop for a Physical AI agent: A deterministic robot-navigation run enters an unsafe region and fails. CoreSense converts the outcome into a structured failure memory containing the scenario, failure mode, situation tags, and recommended intent.
Amazon Bedrock Titan Text Embeddings V2 encodes the memory into a normalized semantic vector. CockroachDB Cloud persists the structured record in JSONB together with a VECTOR(256) embedding. The first agent process stops.
A new process observes a related situation and performs semantic vector retrieval. The recalled avoid_region recommendation is applied through a bounded planner interface. The agent takes a different route and completes the scenario successfully. The demonstration includes a browser-based dashboard that visualizes the baseline failure, recalled memory, evidence fields, and memory-guided successful route. CoreSense does not allow retrieved text to directly control the robot. Memory produces a bounded recommendation, while the deterministic planner remains responsible for the final route.
How we built it
CoreSense is implemented as a Python application with two independent execution phases: record and recall.
Amazon Bedrock： We use Amazon Titan Text Embeddings V2 through Amazon Bedrock to generate 256-dimensional normalized embeddings for failure memories and future situation queries.
The embedding represents semantic context such as: robot and environment type; terrain and situation characteristics; observed failure mode; recommended recovery intent.
CockroachDB Cloud： CockroachDB Cloud provides the persistent agent-memory backend. Each record contains: a stable memory ID; scenario and outcome metadata; failure diagnosis; recommended intent; structured JSONB payload; embedding model provenance; a VECTOR(256) embedding.
A CockroachDB vector index enables cosine-similarity retrieval of relevant memories. Because the memory is stored outside the original Python process, it remains available after the agent stops and restarts.
Agent loop： The public demonstration uses a deliberately small deterministic navigation scenario so the effect of memory can be inspected directly. Without memory, the shortest route crosses an unsafe cell and fails. After the failure has been persisted and recalled, the planner receives a bounded avoid_region recommendation and finds a safe alternative route.
Interface： The dashboard is implemented with HTML, CSS, and JavaScript. It presents: failure and success routes;
the recalled memory; similarity score; failure diagnosis; recommended intent; evidence metadata; the transition from failure to memory-guided success.
The public repository also provides a deterministic local embedding and JSONL backend so reviewers can run the complete control flow without cloud credentials. The submitted cloud path uses Amazon Bedrock and CockroachDB Cloud.
Challenges we ran into
An in-memory Python object does not constitute persistent agent memory. We separated failure recording and memory recall into two commands so that success requires a new process to retrieve the previously persisted record.
Making semantic memory actionable without making it unsafe： A retrieved memory may be irrelevant, stale, or incorrect. We therefore avoid executing arbitrary retrieved content. CoreSense maps the recalled record to a small, inspectable recommendation vocabulary, such as avoid_region, and lets a deterministic planner decide how to apply it.
Aligning embedding and database representations： Amazon Bedrock returns numerical embeddings, while CockroachDB expects a correctly dimensioned vector representation. We added normalization, finite-value checks, dimension validation, and deterministic vector serialization before database insertion and querying.
Designing an honest, reproducible demonstration： The navigation fixture is explicitly labeled as synthetic. It is not presented as a real robot trial or as evidence of large-scale fleet deployment. Its purpose is to isolate and demonstrate the causal memory loop clearly: the same agent and scenario fail without recalled memory and succeed after the relevant memory is retrieved.
Protecting credentials and infrastructure Database URLs and AWS credentials are loaded from environment variables and excluded from version control. The public repository contains .env.example, setup instructions, and a local fallback rather than deployed secrets.
Accomplishments that we're proud of
Engineering a "Database-Native" Pipeline: Successfully transforming raw binary sensor logs into queryable, semantically indexed events.
3D Spatial-Temporal Visualization: Creating an intuitive, high-performance R3F dashboard that renders synchronized multi-modal data in real-time.
Scalable Architecture: Building a system that natively supports distributed deployment, proving that memory for robots can be as scalable as any cloud-native enterprise application.
What we learned
We learned that useful agentic memory requires more than saving conversation history or raw logs.
A physical agent needs memory that is: persistent across processes; semantically retrievable; bound to structured outcomes; associated with evidence and model provenance; filtered before use; connected to a measurable change in behavior.
CockroachDB is valuable because the structured memory and vector embedding live in one transactional database. Amazon Bedrock provides a managed semantic representation layer that allows the agent to retrieve conceptually related experiences rather than depending on exact keyword matches. The most important lesson was that retrieval alone is not the final result. Agentic memory becomes meaningful only when a recalled experience changes the next decision in a controlled and inspectable way.
What's next for CoreSense: Distributed Memory Fabric for Physical AI
Larger and more ambiguous memory banks： We will extend the evaluation from a single controlled failure memory to diverse memories covering different terrains, robots, failure modes, and recommended intents. This will include related, unrelated, and ambiguous queries.
Visual episodic memory： The next research version will associate RGB, depth, pose, and scene context with failure memories. This will allow a robot to recall a prior failure when it visually encounters a related physical situation.
Public robotics datasets： We plan to evaluate visual and spatial memory using publicly available robotics datasets, including quadruped navigation trajectories under challenging environmental conditions.
Real-robot pilot： A future supervised pilot will connect CoreSense as a sidecar memory service to a laboratory robot. CoreSense will record and retrieve execution outcomes but will not replace the robot’s existing safety controller.
Multi-agent and fleet memory： Future work will investigate how multiple robots can contribute to and retrieve from a shared memory space while maintaining access controls, applicability boundaries, and deployment provenance.
Edge and cloud coordination： We plan to retain latency-critical safety and control functions at the edge while using CockroachDB Cloud and managed embedding services for durable, searchable, cross-session experience.
Log in or sign up for Devpost to join the conversation.