Database Accelerator

Bridges backend and database to unlock concurrency and connection benefits while maintaining eventual consistency. Benefits large projects on a single database saving cost and complexity.

Comment

Inspiration

Having built many projects and handling projects with huge databases, one of the pain points has consistently been database connection limit. Solutions range from having replica, sharding, and scaling. But that option only applies to the ones who can afford. For most of the startups and projects to be deployed into the market, infrastructure (servers) is essential and the most financially consuming resources. From experience, the frontend and backend rarely hit their limits, but database connection almost always does. So, I needed a solution that could unblock this limit without the complexity of replication or sharding and ultimately saving cost of management, operations, and infrastructure.

What it does

A single binary (small) runs. It sits between the database and the backend. The backend connects to it like any other database (No changes for existing infra) and enjoys the benefit of being able to have approximately x8 connections to the database that it would have by directly connecting it to the database itself. The performance comes form

How we built it

Step 1: Layout intention, aim, tech stack, basic plans, and goals to Sol 5.6 on extra high thinking and ask it to check feasibility and high-level plan Step 2: Once settled, ask Sol 5.6 on extra high thinking to write detailed plans in a structured and organized way (Per task basis) Step 3: Ask Sol 5.6 on extra high thinking to start implementing it. [Token exhaused] Step 4: Asked claude to fix Github CI to generate binaries.

Challenges we ran into

Token Limits, and UI design. Had provided Design md file for anti-slop design patterns but it did not follow consistently. On third iteration of UI revamp, it was usable and somewhat acceptable. Github CI actions did not go through cleanly with the generated action codes.

Accomplishments that we're proud of

A working prototype since the project has very strict requirements of behaving exactly like a real database yet maintaining performance gain as intended.

What we learned

Sol 5.6 is extremely good at providing realistic approaches to ideas. Example, sqlite in-memory database was also suggested to act as a bridging layer for the same purpose, which the LLM rejected as a bad idea. Secondly, the idea looks easy but its not quite simple given that transactions require atomicity and its extremely difficult especially being in a middle layer. Some compromises in latencies are required to have advantages over other parameters. However, the benefit outweighs for the current goal.

What's next for Database Accelerator

Expand support to as many databases as possible especially postgres. Fune tune more to have really compelling gain rather than just fair advantage. Enhance GUI. Aim of at least x16 with no noticeable compromises.

Built With

Share this project:

Updates