NoteNet

Inspiration

We built NoteNet because we recognized the struggle students face with scattered lecture materials and handwritten notes. We saw firsthand how disconnected information can hinder learning, so we leveraged modern AI to transform raw inputs into a structured, visual map of knowledge. Our goal was to create a tool that makes studying more efficient and insightful.

What it Does

NoteNet converts lecture audio and note images/PDFs into an interactive knowledge graph. We use Whisper to transcribe recordings and OCR tools (such as Tesseract or PyPDF2) to extract text from images and PDFs. Then, powered by GPT-4, we extract key topics and relationships, mapping them into nodes and edges. This graph not only provides a visual representation of the material but also highlights gaps in the notes, enabling targeted review.

How We Built It

Backend:
- We implemented robust file processing: transcribing audio with Whisper and extracting text from images/PDFs using OCR.
- Our NLP pipeline, powered by GPT-4, extracts key topics and relationships from the text.
- We developed a caching system using file hashes to update our vector and document stores efficiently, avoiding redundant processing.
- We built a FastAPI backend to handle file uploads and serve the processed data.
Frontend:
- We created a responsive React-based interface (using Vite) that displays the interactive knowledge graph using React Flow 2D. This allows users to interact with nodes and view detailed note passages when a node is clicked.

Challenges We Ran Into

Data Diversity: Handling various input types—audio, handwritten notes, PDFs—meant dealing with inconsistent quality, from OCR errors to low-quality audio.
Integration Complexity: Seamlessly connecting multiple AI services (Whisper, GPT-4, and vector stores) required careful planning to ensure smooth data flow.
Edge Creation Logic: One of the toughest challenges was designing the logic for creating edges between nodes in the knowledge graph. After experimenting with different methods, we leveraged an LLM to break down relationships into graph edges. While this approach works, it's an area we plan to refine further.

Accomplishments That We're Proud Of

Rapid Prototyping: As a team, we built a working prototype in just 20 hours that covered the entire pipeline—from file upload and preprocessing to knowledge graph visualization.
Seamless AI Integration: We successfully combined transcription, OCR, and NLP to extract meaningful insights from diverse data sources.
Interactive Visualization: Our knowledge graph not only maps out topics but also lets users drill down into specific areas of their notes using React Flow 2D.

What We Learned

The need for a flexible and reliable caching mechanism to manage evolving data.
The value of iterative design—continuous feedback was essential to refining functionality and user experience.
That while AI can extract insights from unstructured data, human oversight remains crucial to validate and refine those insights.

What's Next for NoteNet

Moving forward, our next steps are:

Enhanced Relationship Mapping: Refine our extraction algorithms and edge creation logic to capture more accurate and nuanced connections between topics.
Expanded Input Support: Add support for additional file formats and improve preprocessing to handle even messier, real-world data.
Time Improve processing Latency