Smart Bookmarks

Inspiration

Bookmarks are a reflection of our likes and preferences when it comes to information sources on the web. They provide shortcuts for quick access to content and resources we have found useful or interesting online. However, many of us will agree that the user-experience of the native Bookmarks feature in today's browsers have significant room for improvement. For instance, in the Chrome Browser it is cumbersome to find the best folder to add a new bookmark to. Moreover, it allows us to add a bookmark to exactly one location within the folder hierarchy. However, the content on a website or webpage may span multiple distinct topics. Wouldn't it be great if it were possible to add an entry to multiple bookmark folders simultaneously, where each folder pertains to a specific topic relevant to the content on the site/webpage. Now, considering the current search experience, again many will agree that it is not easy to find a previously bookmarked page, especially if there are a large number of entries to search over. Unless you can remember the exact keywords you used in the title of the entry, a bookmarked page is almost impossible to locate quickly when you are looking for it!

Now, with the easy access to LLMS and intelligence (AI) to understand website content at our fingertips (via Generative AI APIs embedded in Chrome), we are excited to see if we can create a smarter bookmarking experience for everyone.

What it does

We created a 'Smart Bookmarks' Chrome extension with the following two key capabilities:

'Smart Add' feature: When adding a new bookmark entry, the extension generates a summary of the page contents and suggests multiple keywords/tags that better represent the contents and topics described on the page. The summary text provides a quick overview to the user while browsing the bookmarks later on. The tags help in better categorization and improve search and retrieval quality.

'Smart Search' feature: This enables better searching and retrieval experience over the bookmarked pages. Instead of a simple keyword-match based search, 'Smart Search' supports natural language queries and semantic search to retrieve more meaningful matches in response to a user-query.

How we built it

We created the 'Smart Bookmarks' Chrome Extension using the Chrome built-in (Gemini Nano) Summarizer and Prompt APIs. The Summarizer API helps generate a short quick-read summary of the contents of the webpage being bookmarked. The Prompt API is used to generate the keywords that are used to improve the quality of search retrieval. We used a cloud-based vector database (Qdrant.io) to store this data (keywords and summary text) to support the natural language-based smart search capability. We used Gemini's embedding function to compute the embeddings for the content being stored in the Vector Database and subsequently when querying & retrieving (i.e., searching) the database. So, in short, we took a Hybrid approach, where we used some local (Gemini built-in AI APIs) and some cloud-based services (Gemini AI for embedding function and Qdrant vector database for vector search) to deliver all the capabilities. Qdrant's search-as-you-type with fully semantic search backend makes the experience as seamless as regular text/keyword searches that we all are used to today. So, by taking a hybrid approach we tried to hit the right balance between performance and quality.

Challenges we ran into

We had some initial challenges in activating the Summarization and Prompt APIs. They were resolved fairly easily. The device capabilities were a bit of an issue as well and we had to upgrade to a more powerful machine to develop and test the Chrome extension. This will also inform our architectural choices as we keep adding features to the extension. We want the broadest set of Chrome users to be able to use this extension, and not be constrained by the limitations of their hardware.

Accomplishments that we're proud of

We are proud of the fact that we started this project barely a couple of weeks back (mid-November) and in such a short time were able to deliver a great first version of a usable product. The Gemini APIs provided just the right set of functionalities for us to implement two core capabilities that illustrate the improved user-experience of the smart bookmarks extension and of our vision.

What we learned

We saw first-hand, the power of LLMs and Generative AI technology, and how these can help create entirely new user-experiences when it comes to browsing & searching for information on the web. We learnt the benefit of having local (Gemini Nano) models and also, some of their limitations. Certain optimizations will be required to make sure response times are acceptable to users as we continue to build more intelligence into the application. We have many ideas for new features (some listed below) and about how to achieve the right balance between performance and quality, which will be the key.

What's next for Smart Bookmarks

Here are some of the features we are planning to build in the near future:

• Enrich the entries of previously bookmarked pages, i.e., summarize content of previously bookmarked pages (that have not been vectorized) and enable smart search functionality to retrieve matches from this set of prior bookmarked pages as well.
• Support multiple-folder memberships for bookmarked items.
• Improve visualization and browsing of bookmarked items using richer graphical representations, such as embedded images, knowledge graphs and topic-maps.
• Associative, semantic search -- make recommendations for new content on the web based on previously bookmarked content (for example, by using content from multiple bookmarked-pages and issuing them as context in new Gemini Search queries, etc.).
• Better information extraction, summarization and indexing of multimedia content from bookmarked pages.
• Enhanced privacy/security along with cloud backups and cross-device syncing capabilities.
• Better summarization and search over content-heavy pages and dynamic websites -- for example, large embedded PDF documents and news websites.
• Fixing errors in auto-generated summary and metadata; adding personal notes to bookmarks metadata.

Built With

• css
• gemini
• html
• javascript
• prompt-api
• qdrant-vector-db
• summarize-api