Browsing Assistant

Give a patient and tech-savvy assistant to everyone struggling to use a computer

Comment

Introducing Browsing Assistant — a patient and tech-savvy browsing assistant for everyone struggling to use a computer! 🚀

Inspiration

I want to talk to you about my grandmother. She’s a wonderful person, but she doesn’t use the Internet. She doesn’t understand computers, and current interfaces fail her.

Consider that many essential services, such as shopping, banking, or health care, have basically moved online over the last decade. For my grandmother, this is not much of a problem, my grandfather is enough of an Internet user for both of them. He’s even purchased access to several LLMs!

But she is not an isolated case. According to Pew Research, a quarter of people over 65 do not use the Internet in the US alone. That’s about 15 million people, twice the population of the Bay Area.

We believe that it is because most websites are too complex to be effectively used by the more technologically challenged people, and modern browsers such as Chrome and Firefox do not help at all in this respect. Instead of focusing on what matters most to the user, they revel in the wealth of information they profusely display, with little concern for those too dazzled to effectively manipulate it.

This is not only a problem for senior citizens: in this, the websites also fail visually-impaired people – well over 12 million in the US alone.

This is about 10 percent of the US population that the Internet age threatens to leave behind.

What it does

Instead of facing the internet themselves, users of the Browsing Assistant are helped by Claude. Claude talks to them through simple text or through speech generation, and users can simply talk to it back. Instead of having to face large web pages, Claude makes the users' experience simpler by presenting few key elements per page as well as candidate actions which are the most relevant to the goal the user has expressed.

With its large context window, Claude can read the HTML code of a webpage and extract its most important features, making it a helpful guide through the plentiful, but unstructured information. Moreover, by recursively summarizing them, we keep previous pages in context. Thus Claude remembers what happened earlier in the conversation, so that the interaction with Claude looks like the interaction with a tech-savvy friend.

How we built it

We ask Claude to analyze the contents of every webpage. Claude then shows a summary of its contents and suggests a few natural actions that could be taken. When the page is a list of results of a Google search, it also displays the first few ones as suggestions. When the website contains a long list of possible actions – in another display of user-unfriendliness that we aim to correct – it asks for the user’s desire and ranks the actions accordingly, bringing the closest possibility to the user’s attention. For now, we've built a simple Python demo which demonstrated that Claude can understand web pages and guide the user in the vast World Wide Web, and we will expand this prototype to a complete Browsing Assistant.

Challenges we ran into

We believe in giving you the ultimate browsing experience without compromise. For those moments when time is of the essence, Claude Instant takes the lead, navigating you swiftly through the digital landscape. And when accuracy is paramount, Claude 2 emerges as your trusted companion, leaving no web element unturned.

Accomplishments that we're proud of

To the best of our knowledge, Browsing Assistant stands tall as the world's FIRST fully LLM (Language Model-based) accessible browser. A true game-changer, it empowers visually impaired and elderly users to navigate the vast internet landscape with confidence and ease, eradicating any fear of being considered technologically illiterate ever again.

What we learned

At the heart of Browsing Assistant's groundbreaking achievements lies our first and foremost lesson: discovering Claude's incredible ability to comprehend not just text but also HTML pages in a zero-shot manner. This revelation has empowered us to deliver a browsing experience like no other, seamlessly bridging the gap between human language and web elements. Equipped with this powerful understanding, we've learned that for the simplest queries, smaller models can more than suffice, unlocking blazing-fast performance without compromising accuracy. With Claude by our side, we've redefined what's possible in web browsing, making accessibility and efficiency accessible to all.

What's next for Browsing Assistant

In the upcoming versions, we're supercharging Browsing Assistant to handle a whole array of standard items found on web pages. From search forms that seamlessly connect you to the information you seek, to authentication pages that keep your online experiences secure, we're revolutionizing accessibility one step at a time!

But that's not all. Say hello to a new era where images and videos become fully integrated into your browsing journey. Whether it's captivating visuals or enlightening videos, Browsing Assistant will effortlessly bring them to your fingertips.

Built With

Share this project:

Updates