Inspiration

As AI applications scale, developers increasingly need robust, isolated environments that can handle both traditional caching and complex vector searches. We were inspired to build a solution that eliminates the need to run separate databases for key-value storage and vector similarity search. We wanted to create a lightweight, highly concurrent memory engine that provides true per-tenant isolation without sacrificing the speed required by modern LLM-driven applications.

What it does

DBX is an open-source, per-tenant memory engine that seamlessly combines high-speed key-value storage with an integrated HNSW (Hierarchical Navigable Small World) vector index. By supporting RESP2 compatibility, DBX acts as a drop-in replacement for existing Redis-like workflows while extending capabilities into the AI space. It ensures data durability through Write-Ahead Logging (WAL) and allows developers to maintain isolated memory spaces for different tenants or microservices.

How we built it

We engineered DBX entirely in Go to take full advantage of its powerful concurrency model and low-latency networking capabilities. The core architecture relies on custom-built data structures for the in-memory key-value store and the HNSW vector index. We implemented a custom Write-Ahead Logging mechanism to guarantee data persistence across restarts. Finally, we built a protocol parser to ensure full RESP2 compatibility so that standard clients can communicate with DBX natively.

The Math Behind the Vector EngineAt the core of DBX’s AI capabilities is the Hierarchical Navigable Small World (HNSW) algorithm. HNSW relies on multi-layered proximity graphs to achieve high-dimensional similarity search in $\mathcal{O}(\log N)$ time, bypassing the immense computational cost of brute-force nearest-neighbor searches.1. Vector Distance MetricsTo determine how "similar" two tenant data points are, DBX calculates the spatial distance between their high-dimensional vector representations. We evaluate this using two primary mathematical metrics:Euclidean Distance (L2): Measures the straight-line distance between two vectors $p$ and $q$ in an $n$-dimensional space.$$d(p, q) = \sqrt{\sum_{i=1}^{n} (q_i - p_i)^2}$$Cosine Similarity: Measures the cosine of the angle between two non-zero vectors $A$ and $B$. This is especially critical for semantic text embeddings where the direction of the vector holds more meaning than its magnitude.$$S_C(A, B) = \frac{A \cdot B}{\Vert{}A\Vert{} \Vert{}B\Vert{}} = \frac{\sum_{i=1}^{n} A_i B_i}{\sqrt{\sum_{i=1}^{n} A_i^2} \sqrt{\sum_{i=1}^{n} B_i^2}}$$2. Layer Assignment DistributionHNSW constructs a hierarchy of graphs. The bottom layer (Layer 0) contains every ingested vector. As you move to higher layers, the number of vectors decreases exponentially, creating fast navigational "highways" for the search algorithm. When a new vector is inserted into a DBX tenant space, its maximum layer $l$ is determined stochastically using an exponentially decaying probability distribution:$$l = \lfloor -\ln(U) \cdot m_L \rfloor$$Here, $U$ is a uniformly distributed random variable $U \sim \text{Unif}(0,1)$, and $m_L$ is a normalization factor defined as $m_L = 1 / \ln(M)$, where $M$ is the maximum number of connections allowed per node.3. Greedy Search HeuristicDuring a vector query, the DBX engine begins at the topmost layer's entry point. It traverses the graph using a greedy routing approach, evaluating the distance function $d(x, q)$ to find the neighbor node $x$ closest to the query vector $q$. Once a local minimum is reached at layer $l$, the search drops down to layer $l-1$, using that local minimum as the new entry point. This descent continues until it converges on the absolute closest neighbors in Layer 0.

Challenges we ran into

Building a custom database engine from scratch presented several major hurdles. One of the biggest challenges was managing Go's garbage collection overhead during high-throughput vector ingestion and search operations. Ensuring thread-safe, lock-free (or minimal-lock) access to the memory spaces for per-tenant isolation while maintaining microsecond-level latencies required extensive profiling and optimization. Additionally, accurately implementing the HNSW algorithm and seamlessly syncing it with the WAL for crash recovery was highly complex.

Accomplishments that we're proud of

We are incredibly proud of successfully integrating an HNSW vector index directly alongside a traditional key-value store within a single, unified engine. Achieving stable RESP2 compatibility is another major win, as it drastically lowers the barrier to entry for developers wanting to test DBX using their existing tools. Finally, delivering a fully functioning, open-source per-tenant architecture in Go feels like a massive technical milestone.

What we learned

We gained deep, hands-on experience in low-level database engineering. This project pushed us to master Go's concurrency primitives, memory management, and socket programming. We also developed a profound understanding of vector search mathematics, specifically the inner workings of HNSW graph construction and traversal, as well as the rigorous requirements of designing a reliable Write-Ahead Log for durability.

What's next for DBX-Database Extreme

Our immediate next steps involve expanding our protocol support to include RESP3 and adding more complex data structures. Long term, we plan to implement distributed clustering to allow DBX to scale horizontally across multiple nodes, and we want to introduce hardware acceleration (like SIMD instructions) to push our vector search speeds even further.