OpenVidhya

AI driven research paper finder and summarizer.

Comment

Inspiration

  1. Fragmented Research Landscape: Searching for academic papers meant jumping between a dozen different platforms — arXiv, PubMed, Google Scholar, Semantic Scholar — each with different interfaces, paywalls, and inconsistent quality signals.
  2. No Unified Intelligence Layer: Existing tools simply returned raw results with no context about what to read first, what's relevant to your skill level, or how papers connect to each other.
  3. Access Inequality: Open-access research exists across the web, but discovering and reading it without hitting paywalls or broken PDF links remained unnecessarily difficult.
  4. AI Was Underutilized: While LLMs were powerful enough to reason about research, no tool was using them to have a conversation with the user — understanding their background, goal, and expertise before surfacing results.

What it does

  1. Searches 12 databases simultaneously — arXiv, PubMed, Semantic Scholar, OpenAlex, CORE, and more — aggregating and deduplicating results in real time.
  2. Talks to you first — an AI chat assistant (powered by LLaMA 3.3 70B via Groq) asks about your research goal, expertise level, and interests before running a search.
  3. Curates and reranks results — the AI personalizes the paper list based on your context, not just keyword matching.
  4. Summarizes any paper at Beginner, Intermediate, or Expert level so you can quickly decide if it's worth reading.
  5. Generates a reading roadmap — a suggested order to read the top papers for maximum understanding.
  6. Reads PDFs in-app — open-access papers load directly inside the platform without leaving the page.
  7. Saves your research — a personal library to store papers with notes, tags, and collections behind a secure login.
  8. Maps global research — an interactive world map visualizing publication density across countries and scientific domains.

How we built it

  1. Backend: Built a FastAPI server in Python that runs 12 academic API fetchers in parallel using asyncio.gather(), merges and deduplicates results by DOI and title, then applies a relevance ranking algorithm.
  2. AI Layer: Integrated Groq's API to run LLaMA 3.3 70B for conversational query refinement, personalized reranking, multi-level paper summarization, and reading roadmap generation.
  3. Frontend: Built a single-page application in vanilla HTML, CSS, and JavaScript — no framework — with a dark/light theme, glassmorphism design, and an OpenLayers interactive world map for the Research Atlas.
  4. Database: Used SQLite via SQLAlchemy ORM for local persistence — storing user accounts, saved papers, notes, collections, and annotations with JWT-based authentication.
  5. PDF Proxy: Built a backend streaming endpoint to bypass browser CORS restrictions and serve open-access PDFs directly inside an in-app iframe reader.
  6. Infrastructure: Containerized with Docker + Nginx as a reverse proxy. Linux desktop integration via bash launch scripts and a .desktop shortcut.

Challenges we ran into

  1. CORS Blocking PDFs: Browsers blocked direct iframe loading of external academic PDFs. We solved this by routing the PDF byte stream through a backend proxy endpoint.
  2. API Reliability at Scale: Querying 12 external databases simultaneously made the system vulnerable — if one source timed out or went down, it could break the entire search. We wrapped each fetcher in independent timeout and exception handlers so partial failures are silently skipped.
  3. Duplicate Results: The same paper often appeared from multiple sources (e.g., a Nature paper returning from both Crossref and Semantic Scholar). We built deduplication logic matching on DOI first, then normalized lowercase titles.
  4. Port Conflict on Restart: Relaunching the server after a crash left orphaned processes occupying port 8001. We added automatic PID tracking and port-clearing logic to the startup script.
  5. LLM Response Parsing: LLaMA 3.3 sometimes wrapped its JSON response in conversational filler text. We used regex to reliably extract the JSON block regardless of surrounding text.

Accomplishments that we're proud of

  1. 12 APIs in True Parallel: Successfully aggregating and deduplicating results from 12 completely different academic databases simultaneously — each with different schemas, auth methods, and response formats — into one clean, unified result list.
  2. Conversational AI Search: Instead of a simple keyword box, we built a full multi-turn AI conversation that understands your expertise level, research goal, and time period before running a search — a genuinely intelligent research assistant.
  3. Zero Framework Frontend: The entire single-page application — routing, state management, animations, dark/light themes, map integration, PDF docking — was built in pure vanilla HTML, CSS, and JavaScript with no React or Vue. Clean and fast.
  4. Production-Ready Infrastructure: Shipped a complete Docker + Nginx deployment stack with rate limiting, security headers, health checks, and CI/CD workflows — ready to hand to a client and deploy on any Linux server in under 20 minutes.
  5. Fully Self-Contained Offline App: The entire platform — backend, frontend, database, and launcher — runs locally with a single bash start.sh command, no internet required beyond API calls.

What we learned

  1. Parallel Async Design is Non-Negotiable at Scale: Sequential API calls would have taken 30+ seconds per search. Learning to orchestrate asyncio.gather() with independent error handling per source was the single biggest performance unlock.
  2. LLMs Need Guardrails: Raw LLM responses are unpredictable — they add filler, change JSON structure, or hallucinate fields. We learned to always validate and extract structured output defensively using regex and fallback logic.
  3. Browser Security is a Real Constraint: CORS, iframe sandboxing, and mixed-content policies are not just warnings — they block real features. Learning to route sensitive content through a backend proxy was a critical architectural lesson.
  4. Rebranding is an Architecture Audit: Renaming a project taught us that good codebases should centralize config values (database names, service names, ports) in one place rather than scattering them across dozens of files.
  5. Shell Scripting is Underrated: Writing robust bash scripts for process management, port clearing, venv activation, and desktop integration made the app feel truly production-grade and client-ready — not just a developer prototype.
  6. Documentation is Part of the Product: Maintaining a comprehensive README.md, DEPLOY.md, and Roadmap.md alongside the code made the difference between a project and a shippable product.

What's next for OpenVidhya

  1. Semantic Search: Integrate a local vector database (Chroma or Qdrant) with text embeddings so users can search their saved library using natural language — not just keywords.
  2. Local LLM Support: Add Ollama integration so the AI features work completely offline without depending on the Groq cloud API — making it truly air-gapped and privacy-first.
  3. Enhanced PDF Annotations: Native in-app highlighting, drawing, bookmarking, and margin notes directly on the PDF reader — saved and synced to the user's library.
  4. Multi-User & Collaboration: Migrate from SQLite to PostgreSQL and introduce shared research collections — allowing teams and study groups to collaborate on the same paper library.
  5. Mobile-Responsive UI: Redesign the frontend layout to work seamlessly on tablets and mobile devices, expanding OpenVidya beyond the desktop.
  6. CI/CD & Auto-Deployment: Fully automate Docker image builds, test runs, and GitHub releases using the existing GitHub Actions workflow for zero-friction client deployments.

Built With

Share this project:

Updates