Inspiration

Every AI search starts by finding the right documents among millions. At that scale, standard vector search needs tens of GB of memory and seconds per query. We wanted to make it cheap without losing accuracy.

What it does

BitRAG makes search 32x smaller and up to 17x faster with the same accuracy. Our live demo searches 8.8M web passages, 123k photos, NIH medical data and SEC filings, with voice search (ElevenLabs) and a check that says when your answer isn't in the data.

How we built it

We embed every item with open-source models on an AMD MI300X GPU, compress each vector to 1 bit per number, scan with Hamming distance, then re-score the top 100 with 8-bit vectors. Backend: FastAPI + FAISS.

Challenges we ran into

Embedding 8.8M passages fast (we parallelized it to 44 min), slow image preprocessing (39 → 650 photos/sec), and search returning wrong-but-close results when the answer didn't exist.

Accomplishments that we're proud of

On 8.8M documents: same recall, 2.14 s → 126 ms per search, 36 GB → 1.1 GB of memory.

What we learned

Compression alone loses accuracy. The rerank wins it back, because the bits only need to shortlist, not rank. The gains grow with scale.

What's next for BitRAG

Billion-scale search with approximate indexes, and an upload-your-own-data mode for hospitals, banks and schools.