Inspiration
We wanted to build an AI app that can read documents and answer questions, but without using expensive cloud servers. Using open-source tools and lightweight models, the goal was to make everything based on specific papers, not the whole internet.
What it does
- Lets users ask questions based on uploaded documents.
- Searches documents smartly using ChromaDB.
- Runs everything inside Docker containers.
- Show where the AI got its answers (citations).
How I built it
- Frontend: Streamlit (simple web app)
LLM Backend: Ollama (self-hosted AI models)
Document Search: ChromaDB
Containers: Docker and docker-compose
Hosted: AWS EC2
Used Python libraries like requests, chromadb, and langchain.
Challenges I ran into
Some models were too big and crashed because of limited memory.
Needed to make sure all services (Ollama, ChromaDB) were running before Streamlit started.
Figuring out the best way to chunk and search documents took some testing.
Accomplishments that I'm proud of
Built a full AI app that runs completely offline.
Made it modular and easy to update or expand.
What I learned
- How to connect LLMs and databases.
- How to deal with memory limits when using big AI models.
- How to make an app more reliable using Docker and error handling.
What's next
- Let users upload new documents from inside the app.
- Deploy it online for easy access.
- Allow switching between different AI models.
Built With
- amazon-web-services
- chromadb
- docker
- ollama
- python
- streamlit