A zero-DDL, crash-safe columnar database engine built from scratch in Rust, with an embedded local SLM for fast ingestion and deterministic query compilation.
Why I Built SynapseDB
While working with modern data systems, I noticed that there is still a gap between fast ingestion, flexible schemas, analytical workloads, and AI-powered querying.
Traditional relational databases are reliable and powerful, but they usually require a predefined schema. When incoming data changes frequently, this often means dealing with migrations, ETL pipelines, and additional infrastructure.
Document databases solve part of this problem by allowing flexible JSON-like data. However, when the workload becomes analytical, such as calculating SUM, AVG, or COUNT across a large collection, the row-oriented storage model is not always ideal.
There is also another problem with using vector databases for structured numerical questions. Vector search is useful for finding relevant information, but it is not designed to perform deterministic calculations. Asking an LLM to calculate totals across thousands of records can result in missed records or incorrect numbers.
Finally, adding a cloud LLM to the database query path introduces network latency, API costs, and potential data privacy concerns.
So I started building SynapseDB around a simple idea:
Can a database accept evolving data without requiring DDL, store it durably, perform analytical queries efficiently, and understand natural-language queries without sending the data to a cloud LLM?
That became SynapseDB.
What SynapseDB Does
- Sub-Millisecond Durable Writes
Every incoming record is first written to an append-only binary Write-Ahead Log (WAL).
The WAL uses frame headers, CRC32 checksums, fsync-backed persistence, record-level recovery, and WAL replay after crashes.
The database acknowledges a write only after the record has been durably committed.
The goal is to keep the write path below 1 ms while still providing actual durability rather than simply keeping data in memory.
- Zero-DDL Schema Evolution
SynapseDB does not require a CREATE TABLE statement before data can be inserted.
It can accept JSON, key-value records, semi-structured data, and unstructured logs.
The engine analyzes incoming fields and infers physical types such as Int64, Float64, Utf8, and Bool.
As new fields appear, the schema evolves automatically.
Type widening is also handled internally, so the database does not need a traditional migration every time the shape of incoming data changes.
- Embedded Local SLM
One of the more experimental parts of SynapseDB is the embedded Small Language Model.
Instead of sending database questions to an external LLM API, the model runs locally inside the database process using a CPU-quantized model connected through C/Rust FFI.
For example, a user can ask:
"Show me the total revenue for orders above 5000."
Rather than allowing an LLM to generate arbitrary SQL, SynapseDB uses grammar-constrained decoding with GBNF to generate a structured internal representation.
The resulting AST is then validated against the database catalog before execution.
This makes the natural-language query path deterministic and keeps the actual data inside the database.
- Columnar Storage and Vectorized Analytics
SynapseDB stores analytical data in contiguous columnar chunks instead of keeping everything in a traditional row-oriented layout.
This makes operations such as SUM, AVG, COUNT, MIN, and MAX more suitable for vectorized execution.
The engine also maintains Min/Max zone maps for chunks.
If a query is looking for:
temperature > 80
and a particular chunk contains values only between 20 and 40, the engine can skip that chunk completely.
This reduces unnecessary memory scanning and helps keep analytical queries fast.
- Multiple Ways to Use the Database
SynapseDB currently supports multiple interfaces:
Direct SQL queries Natural-language queries Line-delimited TCP protocol Electron-based desktop studio Next.js web playground
The TCP server currently uses port 8765 for the database wire protocol.
How SynapseDB Is Built
The project is organized as a modular Rust Cargo workspace.
synapse-wal
Handles the durable storage layer.
It provides binary append-only logging, WAL frame headers, CRC32 validation, crash recovery, and WAL replay.
synapse-catalog
Manages the dynamic schema.
It handles schema inference, dynamic fields, type widening, schema metadata, and semantic synonym resolution.
synapse-columnar
Responsible for the columnar storage layer.
It manages typed vectors, contiguous memory chunks, columnar data, zone maps, and Arrow-compatible representations.
synapse-slm
Runs the local Small Language Model.
It handles CPU-quantized inference, C/Rust FFI, GBNF grammar constraints, and natural-language-to-AST compilation.
synapse-engine
This is the query execution layer.
It handles scans, filters, aggregations, vectorized execution, and SIMD-ready execution paths.
synapse-server
Provides the network interface.
It uses Rust, Tokio, async TCP, and a lightweight line-delimited protocol.
desktop/ and web/
The project also includes SynapseDB Studio built with Electron and Vite, along with a web playground built with Next.js and TypeScript. The system can also be containerized using Docker.
Some of the Engineering Problems I Had to Solve
The SLM Was Too Slow for the Write Path
One of the first problems I ran into was that even a small quantized 0.5B model can take around 15–40 ms for CPU inference depending on the hardware and workload.
If the database waited for the model before committing a record, the SLM would become the bottleneck.
So I separated the two paths.
The write path is:
Client ↓ Record ↓ WAL ↓ fsync ↓ Acknowledgement
The SLM processing happens separately:
Incoming Records ↓ Lock-Free Ring Buffer ↓ SLM Worker Pool ↓ Background Processing
This means database durability does not depend on the SLM's inference latency.
Making Natural Language Deterministic
Another problem was letting the LLM directly generate SQL.
A free-form LLM can generate invalid SQL, reference columns that do not exist, misinterpret field names, or produce syntactically valid but logically incorrect queries.
Instead, SynapseDB constrains the model to generate an internal AST.
The flow becomes:
Natural Language ↓ Local SLM ↓ GBNF-Constrained Decoding ↓ Internal AST ↓ Catalog Validation ↓ Query Engine ↓ Deterministic Result
The model is therefore used primarily as a query compiler rather than as the component responsible for calculating the final answer.
Testing Crash Recovery
Durability was another area I wanted to take seriously.
I created crash-recovery tests that intentionally terminate the database process unexpectedly and then restart it.
The recovery process replays the WAL and verifies that committed records are restored without corruption.
The goal is simple:
Write ↓ WAL Commit ↓ Unexpected Crash ↓ Restart ↓ WAL Replay ↓ Recover Committed Records
This helped me verify that SynapseDB is not just a fast in-memory prototype.
AWS and Cloud Architecture
SynapseDB is written in Rust, so there is no garbage collector or managed runtime involved in the core engine.
One of the optimization targets is AWS Graviton.
The architecture can take advantage of ARM64 processors and ARM NEON SIMD instructions for vectorized workloads.
I am also designing the storage layer with a future tiered-storage model in mind.
The long-term architecture looks roughly like this:
Hot Data ↓ In-Memory Columnar Chunks ↓ Segment Compaction ↓ Parquet / Arrow IPC ↓ Amazon S3 ↓ Cold Analytical Storage
This would allow frequently accessed data to remain fast while older data can be moved to inexpensive object storage.
What I Learned Building It
The biggest lesson was that database performance is not just about how fast a query runs.
If a database can process a query in microseconds but loses data after a crash, the performance does not matter much.
Implementing the WAL, CRC validation, fsync-based durability, and recovery logic made that distinction very clear.
I also found that small local models can be useful for very specific database tasks.
Instead of putting a large general-purpose LLM in the critical path, a smaller model can be constrained to produce a limited set of database operations.
That gives the system no API or token cost, no external network dependency, better data privacy, predictable output format, and much smaller inference requirements.
The model does not need to "know everything". It only needs to understand how to translate a user's question into the database's supported operations.
What's Next
There is still a lot I want to build.
Planned improvements include:
Explicit AVX-512 SIMD kernels ARM NEON optimized aggregation kernels Disk-backed segment management Parquet and Arrow IPC persistence GROUP BY ORDER BY Multi-table hash joins Better query optimization Python SDK Go SDK TypeScript SDK More complete SQL support Improved SLM query planning
Project Links
GitHub Repository: https://github.com/Sohan-2001/SynapseDB
License: MIT License
Web Playground: https://synapsedb-nine.vercel.app/
In One Sentence
SynapseDB is my attempt to build a database engine from scratch that combines flexible schema-less ingestion, durable storage, columnar analytics, and local AI-assisted query compilation in a single Rust-based system.
Log in or sign up for Devpost to join the conversation.