Inspiration
I wanted to actually understand what "durable" means in a database, not just read about it. The only way to find out is to build a write-ahead log, make a durability promise, and then try as hard as possible to break that promise.
What it does
KVStore is a crash-safe, dependency-free write-ahead-log key-value store written in Go. Every write is appended to a WAL as a CRC32C-checksummed record, with group commit batching concurrent writers into a single fsync. Two configurable durability policies are supported: always (fsync every write) and everysec (fsync at most once per second). On restart, recovery replays the log and distinguishes a torn tail (an incomplete record from an in-flight write, which is safely truncated) from real corruption (a checksum mismatch on an otherwise complete record, which is surfaced as an error rather than silently ignored). The store also supports point-in-time snapshots and shards its in-memory index behind multiple RWMutexes instead of one global lock, to keep reads and writes from serializing on each other under concurrency.
How I built it
Everything is written with zero external dependencies (stdlib only), which forced writing the WAL format, the CRC32C checksumming, and the recovery/replay logic by hand instead of leaning on a library. The design decisions are tracked as 17 ADRs in the repo — for example, sharding the mutex instead of using one global lock, and an honest limitation where directory fsync is a documented no-op on Windows (NTFS doesn't expose the same guarantee POSIX fsync gives you there).
Challenges I ran into
Proving durability claims instead of just asserting them was the hard part. I wrote a crash-test harness that actually kills the process with kill -9 at a random point (200–999ms) during writes — not a simulated failure, a real one — across both fsync policies for 12 iterations each, so the suite proves recovery works 24 different ways on every run, not just once. Building that harness surfaced more bugs than any amount of manual testing did, including an early case where a torn tail was being misclassified as corruption.
Links
Built With
- go
- key-value-store
- systems-programming
- write-ahead-log
