Inspiration
Every semester, the night before an exam, students do the same thing: they open five years of past papers and try to spot patterns manually. Which topics keep coming back? What appeared three years in a row and then vanished? What showed up last year for the first time?
I was doing exactly this for my own KTU exams when I realized — this is a data problem. Past papers are structured data. Recurring questions are clusters. Year-over-year appearances are time series. The "importance" of a topic is a weighted score, not a gut feeling.
That realization became QPredict.
What it does
QPredict takes a folder of previous-year PDF question papers, extracts every question, groups semantically similar ones into topic clusters, and ranks them by a Historical Priority Score — a transparent, evidence-based metric that tells you which topics have the strongest pattern of appearing in past exams.
It does not predict the future. It makes the past legible.
How I built it
1. PDF parsing
The first challenge was getting clean text out of exam PDFs. Indian university papers (especially KTU) have inconsistent formatting, OCR artifacts, and multi-column layouts. I used PyMuPDF (fitz) for extraction and wrote a custom cleaning pipeline to remove page numbers, barcode strings, and boilerplate headers.
2. Question extraction
This was the hardest engineering problem. KTU papers have no standard numbering — Part A questions are plain lines with no prefix, Part B questions have standalone number lines followed by sub-part labels (a), b)) on separate lines before the actual question text.
I built a line-by-line state machine with a lookahead buffer to handle both layouts:
- Layout 1 — fused:
11 a) Explain transposition cipher... - Layout 2 — split across lines:
11 a) Explain transposition cipher... (7)
3. Embeddings and clustering
Questions are embedded using Sentence Transformers (all-MiniLM-L6-v2) with a TF-IDF fallback. Embeddings are clustered using Agglomerative Clustering with a distance threshold (1 − cosine similarity).
The distance threshold controls granularity — lower values mean stricter, smaller clusters. Users can tune it live in the Dashboard sidebar.
4. Historical Priority Score
Each cluster gets a score $S \in [0, 100]$ computed from four components:
$$S = w_f \cdot F + w_c \cdot C + w_r \cdot R + w_k \cdot K$$
| Component | Formula | Default weight |
|---|---|---|
| Frequency $F$ | $\frac{\text{appearances}}{\max_appearances} \times 100$ | 40% |
| Year Coverage $C$ | $\frac{\ | \text{years with topic}\ |
| Recency $R$ | $e^{-\lambda \cdot \Delta t} \times 100$, where $\Delta t$ = years since last appearance, $\lambda = 0.5$ | 20% |
| Consistency $K$ | $\frac{\text{consecutive year pairs}}{\text{span}} \times 100$ | 15% |
The recency decay means a topic that appeared last year scores $R = 100$, while one that last appeared 5 years ago scores $R \approx 8$.
5. Study Assistant (RAG)
The assistant uses Retrieval-Augmented Generation: uploaded paper questions are chunked, embedded into a FAISS vector index, and retrieved at query time. On Streamlit Cloud (where the filesystem resets between sessions), the index is rebuilt from questions stored in the SQLite database rather than from saved text files.
Challenges
1. KTU paper format diversity No two papers had the same structure. Some had marks in a separate column that appeared after all 10 Part A questions. Some had OCR artifacts that split a single question across three lines. I handled this with noise filters, a lookahead buffer, and a minimum character threshold for what counts as a real question.
2. Streamlit Cloud filesystem Streamlit Cloud's container resets between cold starts — uploaded PDFs and processed text files disappear. I solved this by:
- Using SQLite (in the
/mount/src/persistent path) as the source of truth - Rebuilding the RAG index from database questions when text files are missing
- Ensuring the
data/directory is created at startup, not assumed to exist
3. Duplicate element IDs
Streamlit 1.32+ enforces unique element IDs. When rendering 20+ topic cards in a loop, each with a Plotly radar chart, every chart had the same auto-generated ID and crashed the app. Fixed by passing key=f"radar_{sc.cluster_id}" to each st.plotly_chart() call.
4. Broken Python environment
The project's virtual environment was created from a stripped C:\Python313 installation that had no Lib/ folder — causing ModuleNotFoundError: No module named 'encodings' on every Python call. Diagnosed by checking pyvenv.cfg and the base interpreter's directory structure, then recreated the venv from a working Python 3.14 installation.
What I learned
- How to write a line-by-line state machine for parsing semi-structured text at scale
- How sentence embeddings capture semantic meaning that TF-IDF misses (two questions phrased completely differently about the same concept end up in the same cluster)
- How to design a transparent scoring system — every number the app shows has a formula behind it that a student can read and reason about
- The difference between predicting something and describing its history — and why that distinction matters when you're building something students will rely on before an exam
- Practical realities of deploying ML apps on Streamlit Cloud: filesystem ephemerality, cold-start times with large models, and the importance of database-first architecture
What's next
- OCR support for scanned PDFs using Tesseract
- Per-module scoring — KTU papers have 5 modules; scoring within each module separately would be more useful than subject-level clusters
- Cross-subject patterns — some topics (data structures, networking fundamentals) appear across multiple subjects; surfacing those would help students prioritize shared knowledge
Log in or sign up for Devpost to join the conversation.