AI Backend

Try it yourself at https://github.com/JacopoMadaluni/ai-backend

Inspiration

Backends are means to an end. We write them because we need them to create experiences for people. Often though, it's our application that we really care about, and iterating and prototyping efficiently is key.

I always wondered if AI could not only generate backends, but rather making it entirely obsolete, AI as backend.

I don't have to wonder anymore.

What it does

ai-backend Makes AI your backend. It install as a pair of packages:

• A python server
• A javascript SDK

Once up and running, you can forget the backend even exists by just querying a database via natural language. No need to design endpoints, nor relational databases, so you can focus on prototyping your apps.

This is not a production tool

This is not meant to be a production tool, rather a fast prototyping framework to build experiences without having to write backends and databases.

Production size and speed make the current AI models unsuitable for the purpose, at least for now.

How I built it

ai-backend uses pure LLM capabilities to keep track of a database file on the local file system. For this demo, we used mistral-large-latest. The python package mounts a router to create a mini api for the basic operations, that are forwarded to the AI when required. In order for this to really work, the AI must:

• have your database as context
• return the full database to update local state
• be able to stick to my frontend schema

I achieved this with various means of prompting on the system. There is currently a suite of 3 system prompts ensambled together. Pretty simple.

The API serves as a proxy between the state and the llm:

# Simplified endpoint:
@app.post("/q")
async def root(body: QueryInfo):
    q = body.q
    db_path = get_db_file_path()
    db = get_db()
    updated, db, result = submit(q, db, info_str)
    if updated:
        with open(db_path, "w+") as f:
            f.write(json.dumps(db))

    return { "updated": updated, "db": db, "result": result}

AI Backend can also query extra APIs, and internal SDKS

It's also possible to add mappings to give the AI the ability to configure calls to internal SDKs or APIs. This way I can query from frontend any backend available as long as a mapping exists:

import { query } from "ai-backend";

const res = await query(
    "[MIXPANEL] Give me all emails that signed up in the past week"
);

Challenges we ran into

1. Having AI stick to a schema

To me this was the most important aspect of this project.

When we write backends, the most important aspect of them is the contract / interface they provide. By nature, AI doesn't really provide a strict contract, which is why we end up writing bloated frameworks just to ensure contracts are preserved.

If the AI makes even the slightest schema mistakes, the frontend will stop working, making the whole thing unusable for the purpose it was intended to.

2. Speed

The framework is currently poorly optimized, or not optimized at all. When tested on medium size databases, the speed becomes a real issue.

Accomplishments that we're proud of

1. Interaction with a schema

I am genuinely amazed at how well this works. After tweaking the schema system for a bit, I started noticing that all mistakes were not coming from the AI, but rather from me defining the schema improperly in my frontend apps.

In a twisted sense, using this framework made my frontend schemas really accurate.

2. I can query my internal SDKs via natural language

With just 50 lines of mapping on top of the python sdk, the AI is able to send requests to my company's internal system, all whilst conforming to the schemas.

It was weirdly exciting to be able to talk to my SDK via natural language.

3. A solution to internal tooling

This project made me realise that this ai-backend could very well be a very good choice for internal tooling, which are usually really just API wrappers to internal APIs and SDKs, which usually require a frontend and a backend.

I might actually implement our next interal tool with this.

What we learned

1. AI needs to get faster

AI is currently very slow at this. With small prototypes it's fine, but when the task grows in size, the speed becomes annoying.

2. We don't need to "generate code"

We often take the approach of generating code. But AI will soon be good enough to generate behaviour. It's already good enough for implementing prototypes, and it'll soon be good for production use.

I wouldn't be surprised if the backends of the future will rely on natural language to be queried.

What's next for AI as Backend SDK

1. Finetune a small version of the LLM

The current LLM is slow. Too slow. A good fix for this would be having the large llm generate syntetic examples of schema adherence to train a smaller LLM to reach the same level of performance. I think one could achieve the same result with an LLM more than half smaller.

2. Optimize db injection

Right now the whole database is injected to the LLM as context to have it perform operations. This slows it down, and creates a db size limitation.

A system could be implemented where depending on the user query, it prunes the only necessary tree of the database to minimize context.

Built With

• mistral
• python
• typescript