CockroachDB promises never-failing, consistent data - but at scale, silent memory safety issues are the scariest bugs. A single leak or use-after-free in a distributed node can cascade. I got my inspiration from tools like AddressSanitizer and wanted to bring the concept of shadow memory to CockroachDB: a parallel safety layer that watches every allocation without slowing down the cluster.
ShadowSense is a safety memory layer for CockroachDB.
It creates a 1:1 shadow map of the database's memory space to:
- Detect memory leaks per range and per node in real-time
- Flag unsafe access patterns (use-after-free, double-free)
- Visualize memory hotspots across your 3-node cluster
- Provide a safety report without modifying CockroachDB core
Think of it as a flight data recorder for database memory.
How it was built :
- Spun up a local 3-node insecure CockroachDB cluster for testing
- Built ShadowSense in Go, hooking into allocation paths
- Implemented a low-overhead concurrent hash map as the shadow memory store
- Added lightweight instrumentation to track alloc / free / size / stack trace
- Containerized with Docker for reproducible demos and benchmarking
- CLI that prints live safety analysis while the cluster handles SQL load
Some Issues :
Performance vs. Precision: Shadow memory doubles bookkeeping. Keeping overhead <5% was a fight with Go's GC.
Distributed State: Memory isn't in one place - it's sharded. Aggregating shadow state across 3 nodes without consensus bottlenecks was tricky.
Git/Tooling: Honestly, pushing the repo took longer than building v1. Classic hackathon moment.
What I was trying to accomplish:
- Working shadow memory that runs alongside a live CockroachDB cluster
- Successfully detected simulated leaks and double-frees in our tests
- <10ms overhead per 10k operations in our local benchmarks
- Fully open source and Docker-ready - anyone can
clone and run
What I learned:
- Deep internals of CockroachDB's storage engine (Pebble) and how it manages memory
- How real-world sanitizers use shadow mapping and how to adapt that to databases
- Building low-level tooling in Go without crippling performance
What's next for ShadowSense: CockroachDB Safety Memory
- Grafana + Prometheus dashboard for live shadow map visualization
- Integration with
cockroach debugcommands - eBPF version for zero-code-change production monitoring
- Automatic alerting + PR suggestion when a leak is detected
- Submit as a CockroachDB Labs community tool
Log in or sign up for Devpost to join the conversation.