ReCall

ReCall new tab and history graph view
Popup settings page
History search feature

Inspiration

The inspiration behind ReCall came from an incident when one of our group members could not find a website in his browser history. He was able to describe the website, and its content, and could come up with relevant keywords, but could not remember the exact website link. With the normal search function, he was unable to find the website in his browser history. This inspired us to come up with ReCall, which lets you search for websites in your history by conversing with an agent, alongside a graph view showing related websites in your history.

What it does

ReCall uses a Retrieval-Augmented Generation (RAG) system integrated with a Large Language Model (LLM) to help users quickly locate previously visited websites. Unlike the standard browsing history feature in Google Chrome, ReCall enables users to search their history by inputting specific keywords or descriptions about past sites. Webpage data and metadata are stored as vector embeddings using the IndexedDB. When users perform a query, Cohere's Rerank is utilized to retrieve and rank relevant pages based on semantic similarity. Finally, Google Gemini serves as the conversational LLM, presenting users with ranked results through an intuitive chatbot interface.

How we built it

We built ReCall as a Chrome extension using Plasmo's framework alongside React and Tailwind CSS, developing a responsive, user-friendly interface. The backend utilizes IndexedDB to store webpage content in the form of vector embeddings, enabling an efficient semantic search through these processes:

Dynamic Content Processing We used Typescript to build background workers that analyze and classify webpages in real time so that when a user visits a site, the system extracts keywords, summarizes content, and checks against user-defined preferences.

Semantic Search & RAG Pipeline We utilized Coheres API to generate embeddings for webpage content, while Cohere Rerank optimizes query results by semantic relevance. We then used Google Gemini to process natural language queries to convert user inputs into searchable prompts

Graph Visualization Using React Flow we mapped out websites/keywords as nodes and relationships as edges, creating an interactive graph.

Challenges we ran into

Through the process of developing ReCall we ran into several challenges. These included efficiently capturing and processing dynamic web content for accurate embeddings, optimizing performance, and reducing latency when integrating Cohere’s Reranks and Gemini APIs. Additionally making sure user privacy was maintained while handling sensitive browsing data, proved challenging and required multiple iterations. Overall this project required us to balance the speed and accuracy in the RAG-LLM pipeline, demanding careful tuning, this was further constrained by browser extension limitations with challenges related to data collection. Despite these obstacles, we utilized iterative testing and modular design to help us refine our initial idea into a working solution.

Accomplishments that we're proud of

Successfully integrating Coheres embeddings and rerank with Google Gemini into a seamless workflow and creating a functional RAG-LLM pipeline

Designing an optimized node graph that can handle hundreds of nodes without lag utilizing lazy loading and dynamic clustering, allowing for an intuitive user-focused graph UI

Implementing zero-knowledge storage to ensure that user browsing data remains user-controlled and maintains privacy compliance

What we learned

Through this project, the main thing we learned was how to effectively use Plasmo to develop browser extensions. Additionally, we also learned valuable skills relating to browser extension development such as the strict memory and performance limits. Furthermore, we also learned how to effectively incorporate the RAG-LLM pipeline.

What's next for ReCall

Based on the current state of the project our next functional steps include incorporating these features to further improve user experience and functionality: Multimodal Search: Allowing users to describe images and videos from their search history utilizing Gemini’s vision capabilities Cross Browser Support: Develop Versions of the extension to support other browsers like Firefox and Safari Mobile Companion App/Cross-Platform sync: Develop a mobile app to allow for the storage and sync of search histories from all platforms.