main page of the app
voice interaction

🖊️ Pen and Paper

Transform handwritten notes into searchable, conversational knowledge

🧠 Inspiration

Most of our best thoughts still start on paper — lecture notes, meeting scribbles, or quick sketches. But once written, they’re hard to search, organize, or make use of. I wanted a way to bridge the gap between analog creativity and digital intelligence — turning handwritten ideas into structured, searchable, and conversational knowledge.

⚙️ What it does

Pen and Paper scans handwritten documents, performs OCR to extract text, and builds a searchable knowledge base using Meilisearch.

Then, using Google’s Generative AI, it enables Retrieval-Augmented Generation (RAG) — allowing users to ask natural-language questions and get precise answers grounded in their own notes.

It also integrates ElevenLabs to convert responses into lifelike speech, creating an immersive “talk to your notebook” experience.

🏗️ How we built it

FastAPI backend for managing authentication, file uploads, and OCR pipelines.
Google Gemma 3 4B for accurate handwritten text recognition.
Meilisearch for lightning-fast document indexing and retrieval.
Google Generative AI API for contextual reasoning and RAG responses.
React frontend for a clean, responsive UI with voice interaction using ElevenLabs TTS/STT.

🧩 Challenges we ran into

Handling the variability and noise in handwritten text recognition.
Managing multi-tenant search and access control while keeping it fast.
Combining Meilisearch results with generative AI responses without hallucination.
Audio streaming integration between FastAPI and ElevenLabs for real-time voice output.

🏆 Accomplishments that we're proud of

Built a full-stack system that converts handwriting into interactive knowledge.
Seamlessly integrated Google AI, Meilisearch, and ElevenLabs.
Achieved near-real-time RAG responses and high OCR accuracy on real handwriting samples.
Designed a modular architecture ready for self-hosted or offline deployment.

📚 What we learned

How to combine retrieval and generation effectively for grounded AI responses.
Practical considerations in OCR quality, indexing, and prompt design.
The power of asynchronous APIs and event streaming between FastAPI and React.
The importance of user experience — especially when bridging voice, text, and handwriting.